Nov 03 19:26:12.048135 ip-10-0-133-29 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 19:26:12.436881 ip-10-0-133-29 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:26:12.436881 ip-10-0-133-29 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 19:26:12.436881 ip-10-0-133-29 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:26:12.436881 ip-10-0-133-29 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:26:12.441374 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441291    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:26:12.441374 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441373    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:26:12.441374 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441376    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441379    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441383    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441386    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441388    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441391    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441394    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441396    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441399    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441402    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441405    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441408    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441410    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441413    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441416    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441418    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441421    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441424    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441426    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:26:12.441471 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441429    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441433    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441438    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441441    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441444    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441448    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441451    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441454    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441457    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441460    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441462    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441465    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441467    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441470    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441473    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441477    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441479    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441482    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441485    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:26:12.441945 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441487    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441490    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441493    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441496    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441498    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441501    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441503    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441506    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441508    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441511    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441515    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441519    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441522    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441525    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441527    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441530    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441533    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441536    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441539    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:26:12.442396 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441543    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441546    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441548    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441551    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441554    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441556    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441559    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441561    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441564    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441566    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441569    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441571    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441574    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441576    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441579    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441581    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441584    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441586    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441589    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441592    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:26:12.442859 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441594    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441597    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441599    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441602    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441605    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441607    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441610    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441613    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441616    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441619    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441621    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441628    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441632    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441634    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441637    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441640    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441642    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441645    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441648    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441650    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:26:12.443335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441654    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441657    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441660    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441662    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441665    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441667    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.441670    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442493    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442499    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442502    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442505    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442508    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442511    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442514    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442516    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442520    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442523    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442525    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442528    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442531    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:26:12.443804 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442534    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442537    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442540    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442542    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442546    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442549    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442551    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442554    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442556    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442559    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442562    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442564    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442567    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442569    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442573    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442577    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442580    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442583    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442586    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442588    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:26:12.444300 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442591    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442593    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442596    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442598    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442601    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442604    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442606    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442609    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442612    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442615    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442617    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442620    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442624    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442626    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442629    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442633    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442636    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442640    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442642    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:26:12.444774 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442645    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442647    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442650    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442652    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442655    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442657    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442660    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442662    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442665    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442667    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442670    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442672    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442675    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442677    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442680    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442682    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442685    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442687    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442690    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:26:12.445243 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442693    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442696    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442698    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442702    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442705    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442707    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442710    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442712    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442715    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442718    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442720    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442723    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442726    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442729    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442731    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442734    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442736    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442739    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442741    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442744    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:26:12.445694 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442747    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442750    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442752    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442754    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442757    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442760    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442762    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442765    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442767    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442770    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442772    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442775    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442778    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442780    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.442783    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443300    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443305    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443308    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443311    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:26:12.446215 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443314    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443317    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443320    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443323    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443326    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443329    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443332    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443335    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443338    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443340    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443343    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443346    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443349    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443351    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443354    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443357    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443359    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443362    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443365    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443367    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:26:12.446670 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443370    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443372    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443375    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443378    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443380    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443383    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443386    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443389    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443391    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443394    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443396    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443399    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443402    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443404    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443407    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443409    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443412    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443414    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443418    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443421    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:26:12.447175 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443423    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443426    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443428    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443431    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443433    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443437    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443440    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443443    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443446    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443448    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443451    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443453    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443456    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443459    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443461    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443464    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443466    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443469    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443472    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:26:12.447682 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443476    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443480    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443484    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443487    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443489    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443492    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443494    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443498    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443500    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443503    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443505    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443508    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443511    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443513    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443516    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443518    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443521    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443523    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443526    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443528    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:26:12.448144 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443530    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443533    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443535    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443538    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443540    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443543    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443545    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443548    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443551    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443556    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443558    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443561    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443564    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443567    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443570    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443573    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443575    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443578    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443581    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:26:12.448616 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443583    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443586    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443588    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.443591    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444696    2577 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444708    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444716    2577 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444721    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444725    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444729    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444733    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444738    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444741    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444744    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444748    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444751    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444755    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444758    2577 flags.go:64] FLAG: --cgroup-root=""
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444761    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444764    2577 flags.go:64] FLAG: --client-ca-file=""
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444766    2577 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444769    2577 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444778    2577 flags.go:64] FLAG: --cluster-domain=""
Nov 03 19:26:12.449122 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444781    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444785    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444789    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444792    2577 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444796    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444799    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444803    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444806    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444810    2577 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444813    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444816    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444819    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444823    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444828    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444847    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444851    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444855    2577 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444859    2577 flags.go:64] FLAG: --enable-server="true"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444862    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444869    2577 flags.go:64] FLAG: --event-burst="100"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444872    2577 flags.go:64] FLAG: --event-qps="50"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444875    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444878    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444882    2577 flags.go:64] FLAG: --eviction-hard=""
Nov 03 19:26:12.449646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444886    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444889    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444892    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444896    2577 flags.go:64] FLAG: --eviction-soft=""
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444899    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444902    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444905    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444909    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444912    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444915    2577 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444918    2577 flags.go:64] FLAG: --feature-gates=""
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444922    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444925    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444929    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444932    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444935    2577 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444938    2577 flags.go:64] FLAG: --help="false"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444941    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444944    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444948    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444951    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444954    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444957    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 19:26:12.450252 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444960    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444963    2577 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444966    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444970    2577 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444974    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444977    2577 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444980    2577 flags.go:64] FLAG: --kube-reserved=""
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444984    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444987    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444990    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444992    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444995    2577 flags.go:64] FLAG: --lock-file=""
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.444998    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445001    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445005    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445010    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445013    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445016    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445019    2577 flags.go:64] FLAG: --logging-format="text"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445022    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445026    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445029    2577 flags.go:64] FLAG: --manifest-url=""
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445032    2577 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445037    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445040    2577 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 19:26:12.450800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445044    2577 flags.go:64] FLAG: --max-pods="110"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445047    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445051    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445054    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445057    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445060    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445063    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445066    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445074    2577 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445078    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445081    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445086    2577 flags.go:64] FLAG: --pod-cidr=""
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445089    2577 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445092    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445095    2577 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445098    2577 flags.go:64] FLAG: --port="10250"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445101    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445104    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0fef8e101635c0b98"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445108    2577 flags.go:64] FLAG: --qos-reserved=""
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445111    2577 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445114    2577 flags.go:64] FLAG: --register-node="true"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445117    2577 flags.go:64] FLAG: --register-with-taints=""
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445121    2577 flags.go:64] FLAG: --registry-burst="10"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445124    2577 flags.go:64] FLAG: --registry-qps="5"
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445127    2577 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445129    2577 flags.go:64] FLAG: --reserved-memory=""
Nov 03 19:26:12.451426 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445134    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445137    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445140    2577 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445144    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445147    2577 flags.go:64] FLAG: --runonce="false"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445150    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445153    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445156    2577 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445159    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445162    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445165    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445168    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445172    2577 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445175    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445178    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445181    2577 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445184    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445193    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445197    2577 flags.go:64] FLAG: --system-cgroups=""
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445200    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445205    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445209    2577 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445212    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445216    2577 flags.go:64] FLAG: --tls-min-version=""
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445219    2577 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 19:26:12.452041 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445222    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445224    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445227    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445231    2577 flags.go:64] FLAG: --v="2"
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445235    2577 flags.go:64] FLAG: --version="false"
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445240    2577 flags.go:64] FLAG: --vmodule=""
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445244    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445247    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445405    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445409    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445412    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445415    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445417    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445420    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445422    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445425    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445428    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445430    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445433    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445437    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445440    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:26:12.452658 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445442    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445445    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445447    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445451    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445456    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445460    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445463    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445465    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445468    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445470    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445473    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445476    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445479    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445481    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445484    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445486    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445489    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445491    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445494    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:26:12.453194 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445496    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445499    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445501    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445504    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445506    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445509    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445511    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445514    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445517    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445519    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445522    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445526    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445528    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445531    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445533    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445536    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445540    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445542    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445545    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445548    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:26:12.453639 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445550    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445553    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445556    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445559    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445562    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445564    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445567    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445569    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445572    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445574    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445577    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445581    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445584    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445588    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445591    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445593    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445596    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445599    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445602    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:26:12.454134 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445604    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445607    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445610    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445612    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445616    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445619    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445621    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445624    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445627    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445631    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445634    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445636    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445639    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445642    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445645    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445647    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445650    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445653    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445655    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445658    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:26:12.454593 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445661    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:26:12.455198 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445663    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:26:12.455198 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445666    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:26:12.455198 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445669    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:26:12.455198 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445671    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:26:12.455198 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445674    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:26:12.455198 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445676    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:26:12.455198 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445679    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:26:12.455198 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445681    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:26:12.455198 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445684    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:26:12.455198 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445687    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:26:12.455198 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445689    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:26:12.455198 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445692    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:26:12.455198 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445695    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:26:12.455198 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445698    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:26:12.455198 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.445726    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445762    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445767    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445770    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445773    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445775    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445778    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445781    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445784    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445787    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445790    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445793    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445795    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445798    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445801    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445803    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445806    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445808    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445811    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445814    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:26:12.455556 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445816    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445818    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445821    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445823    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445826    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445828    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445842    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445845    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445848    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445851    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445853    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445856    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445858    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445862    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445865    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445868    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445870    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445874    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445879    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:26:12.456042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445882    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445884    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445887    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445890    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445893    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445896    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445898    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445901    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445904    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445906    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445909    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445911    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445914    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445917    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445919    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445922    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445924    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445927    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445929    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445932    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:26:12.456496 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445934    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445937    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445939    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445943    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445946    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445948    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445952    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445954    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445957    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445960    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445963    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445965    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445968    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445971    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445974    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445977    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445979    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445982    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445985    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:26:12.457080 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445988    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445990    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445993    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445995    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.445998    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446000    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446003    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446006    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446008    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446011    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446013    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446016    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446018    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446021    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446023    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446026    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446028    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446031    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446033    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:26:12.457533 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446038    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446042    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446045    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446048    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446051    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446053    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446057    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446060    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446062    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.446065    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.453146    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.453162    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.453921    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.453927    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.453987    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.453992    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454010    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454015    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454019    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:26:12.458042 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454023    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454026    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454028    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454031    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454034    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454037    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454039    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454042    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454045    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454048    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454051    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454053    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454056    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454058    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454061    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454065    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454071    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454074    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454076    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:26:12.458486 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454079    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454082    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454084    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454087    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454089    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454093    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454097    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454100    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454102    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454105    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454108    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454110    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454113    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454116    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454118    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454121    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454123    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454126    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454129    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454131    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:26:12.458947 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454134    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454137    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454140    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454142    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454145    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454148    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454151    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454153    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454156    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454158    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454161    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454163    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454166    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454169    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454172    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454174    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454177    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454181    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454183    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454187    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:26:12.459423 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454189    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454192    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454194    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454197    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454199    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454202    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454204    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454207    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454210    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454212    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454215    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454217    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454220    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454222    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454225    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454227    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454231    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454235    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454238    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:26:12.459988 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454241    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454243    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454246    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454248    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454251    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454253    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454256    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454259    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454261    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454264    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454267    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454270    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454272    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454275    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454278    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454281    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454283    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454286    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454288    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:26:12.460434 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454291    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:26:12.460901 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454293    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:26:12.460901 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454296    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:26:12.460901 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454299    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:26:12.460901 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.454328    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:26:12.460901 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454465    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:26:12.460901 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454470    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:26:12.460901 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454473    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:26:12.460901 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454475    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:26:12.460901 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454478    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:26:12.460901 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454480    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:26:12.460901 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454483    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:26:12.460901 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454486    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:26:12.460901 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454489    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:26:12.460901 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454491    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:26:12.460901 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454494    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:26:12.460901 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454497    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454499    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454502    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454504    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454507    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454509    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454513    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454516    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454518    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454521    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454524    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454527    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454530    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454532    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454534    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454537    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454539    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454542    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454544    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454547    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:26:12.461328 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454549    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454552    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454554    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454557    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454559    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454562    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454564    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454568    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454582    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454585    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454588    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454591    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454594    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454596    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454599    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454602    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454605    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454608    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454611    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:26:12.461800 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454614    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454617    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454621    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454625    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454629    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454632    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454635    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454638    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454640    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454643    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454645    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454648    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454651    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454653    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454656    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454658    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454661    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454663    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454666    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:26:12.462335 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454668    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454671    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454674    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454676    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454679    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454681    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454684    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454686    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454688    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454691    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454694    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454697    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454699    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454702    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454705    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454707    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454710    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454712    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454715    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454718    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:26:12.462797 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454720    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:26:12.463490 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454723    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:26:12.463490 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454726    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:26:12.463490 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454728    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:26:12.463490 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454731    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:26:12.463490 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454733    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:26:12.463490 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454736    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:26:12.463490 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454739    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:26:12.463490 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454741    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:26:12.463490 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454744    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:26:12.463490 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454747    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:26:12.463490 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454749    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:26:12.463490 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454752    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:26:12.463490 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454754    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:26:12.463490 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454757    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:26:12.463490 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454760    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:26:12.463490 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454762    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:26:12.463905 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.454785    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:26:12.463905 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454817    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:26:12.463905 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454821    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:26:12.463905 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454824    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:26:12.463905 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454827    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:26:12.463905 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454829    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:26:12.463905 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454847    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:26:12.463905 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454852    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:26:12.463905 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454856    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:26:12.463905 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454858    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:26:12.463905 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454861    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:26:12.463905 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454864    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:26:12.463905 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454867    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:26:12.463905 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454869    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:26:12.463905 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454872    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454875    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454878    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454880    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454883    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454885    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454888    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454891    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454893    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454896    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454898    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454901    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454904    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454906    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454909    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454911    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454914    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454916    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454919    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:26:12.464265 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454922    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454924    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454927    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454929    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454932    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454935    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454938    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454941    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454943    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454946    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454949    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454951    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454954    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454956    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454959    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454962    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454964    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454967    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454969    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:26:12.464805 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454974    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454977    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454980    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454982    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454985    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454987    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454990    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454992    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454995    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.454997    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455000    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455002    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455005    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455007    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455011    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455015    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455018    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455021    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455024    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:26:12.465282 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455028    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455031    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455033    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455036    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455038    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455041    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455043    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455046    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455049    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455051    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455054    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455057    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455059    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455062    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455064    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455067    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455069    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455072    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455075    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455077    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:26:12.465823 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455079    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455082    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455085    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455087    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455090    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455092    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455095    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455097    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455100    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455102    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455105    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455108    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455110    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455113    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455116    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:12.455119    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.455172    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.457709    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.458778    2577 server.go:1014] "Starting client certificate rotation"
Nov 03 19:26:12.466349 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.459262    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:26:12.466815 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.459305    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:26:12.466815 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.462134    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:26:12.466815 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.464645    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:26:12.477498 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.477471    2577 log.go:25] "Validated CRI v1 runtime API"
Nov 03 19:26:12.482307 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.482291    2577 log.go:25] "Validated CRI v1 image API"
Nov 03 19:26:12.485280 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.485247    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 19:26:12.489925 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.489902    2577 fs.go:135] Filesystem UUIDs: map[4068e963-221f-442e-a7da-88d1e13886b9:/dev/nvme0n1p3 66f1c584-2d12-4182-8a8b-590d47830b0f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 03 19:26:12.490002 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.489921    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 19:26:12.496003 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.495870    2577 manager.go:217] Machine: {Timestamp:2026-11-03 19:26:12.494397067 +0000 UTC m=+0.343611138 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099833 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f6604d39ed266e68c9437d0ed6480 SystemUUID:ec2f6604-d39e-d266-e68c-9437d0ed6480 BootID:de033993-ff93-41ed-b1c5-62b34ffc7442 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d7:03:40:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d7:03:40:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:d7:d2:43:cc:5a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 19:26:12.496003 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.495994    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 19:26:12.496148 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.496076    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 19:26:12.499009 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.498975    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 19:26:12.499153 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.499010    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-29.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 19:26:12.499219 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.499154    2577 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 19:26:12.499219 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.499164    2577 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 19:26:12.499219 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.499199    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:26:12.499219 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.499219    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:26:12.500508 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.500495    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 19:26:12.502523 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.502505    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 19:26:12.505099 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.505085    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 19:26:12.505224 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.505213    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 19:26:12.505267 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.505247    2577 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 19:26:12.505267 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.505259    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 19:26:12.505343 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.505269    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 19:26:12.505343 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.505279    2577 kubelet.go:405] "Adding apiserver pod source"
Nov 03 19:26:12.505343 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.505300    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 19:26:12.505695 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.505678    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:26:12.509326 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.509296    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 19:26:12.518096 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.518078    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:26:12.519060 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.519049    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 19:26:12.519115 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.519078    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 19:26:12.519234 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.519225    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 19:26:12.519267 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.519240    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 19:26:12.519267 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.519247    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 19:26:12.519267 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.519253    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 19:26:12.519267 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.519259    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 19:26:12.519267 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.519265    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 19:26:12.519981 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.519970    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 19:26:12.519981 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.519980    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 19:26:12.520048 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.519987    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 19:26:12.520048 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.519993    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 19:26:12.520048 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.520002    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 19:26:12.520048 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.520019    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 19:26:12.520850 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.520825    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 19:26:12.520884 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.520854    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 19:26:12.522656 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.522632    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-29.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 19:26:12.525077 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.525063    2577 server.go:1263] "Started kubelet"
Nov 03 19:26:12.525198 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.525145    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 19:26:12.525412 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.525367    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 19:26:12.525465 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.525431    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 19:26:12.525979 ip-10-0-133-29 systemd[1]: Started Kubernetes Kubelet.
Nov 03 19:26:12.527131 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.527006    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 19:26:12.528189 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.528142    2577 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 19:26:12.531541 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.531521    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 19:26:12.534063 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.534039    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 19:26:12.534649 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.534628    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 19:26:12.534752 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.534671    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cgg5n"
Nov 03 19:26:12.534934 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.534911    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-29.us-east-2.compute.internal\" not found"
Nov 03 19:26:12.535605 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.535587    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 19:26:12.535691 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.535662    2577 factory.go:55] Registering systemd factory
Nov 03 19:26:12.535691 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.535681    2577 factory.go:223] Registration of the systemd container factory successfully
Nov 03 19:26:12.535817 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.535598    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 19:26:12.535817 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.535819    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 19:26:12.536004 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.535953    2577 factory.go:153] Registering CRI-O factory
Nov 03 19:26:12.536004 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.535962    2577 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 19:26:12.536004 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.535969    2577 factory.go:223] Registration of the crio container factory successfully
Nov 03 19:26:12.536112 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.536036    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 19:26:12.536112 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.535970    2577 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 19:26:12.536112 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.536057    2577 factory.go:103] Registering Raw factory
Nov 03 19:26:12.536217 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.536142    2577 manager.go:1199] Started watching for new ooms in manager
Nov 03 19:26:12.537412 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.537392    2577 manager.go:319] Starting recovery of all containers
Nov 03 19:26:12.537500 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.537454    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-29.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 19:26:12.537556 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.537531    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-29.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 19:26:12.537650 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.537631    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 19:26:12.544190 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.544166    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cgg5n"
Nov 03 19:26:12.547720 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.537302    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-29.us-east-2.compute.internal.18e49f8b5de7ae7b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-29.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-29.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:26:12.525035131 +0000 UTC m=+0.374249201,LastTimestamp:2026-11-03 19:26:12.525035131 +0000 UTC m=+0.374249201,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-29.us-east-2.compute.internal,}"
Nov 03 19:26:12.551735 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.551700    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 19:26:12.554009 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.553991    2577 manager.go:324] Recovery completed
Nov 03 19:26:12.558168 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.558151    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:26:12.560165 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.560148    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:26:12.560248 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.560183    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:26:12.560248 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.560198    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:26:12.560732 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.560716    2577 cpu_manager.go:226] "Starting" policy="none"
Nov 03 19:26:12.560732 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.560728    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 19:26:12.560827 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.560744    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 19:26:12.560827 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.560765    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:26:12.561522 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.561506    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:26:12.561581 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.561533    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:26:12.561581 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.561549    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:26:12.562786 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.562775    2577 policy_none.go:50] "Start"
Nov 03 19:26:12.562826 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.562791    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 19:26:12.562826 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.562801    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 19:26:12.564814 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.564798    2577 policy_none.go:44] "Start"
Nov 03 19:26:12.616539 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.613411    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 19:26:12.616539 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.614619    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 19:26:12.616539 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.614642    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 19:26:12.616539 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.614665    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 19:26:12.616539 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.614710    2577 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 19:26:12.618811 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.618789    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:26:12.636042 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.636023    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-29.us-east-2.compute.internal\" not found"
Nov 03 19:26:12.664522 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.664475    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 19:26:12.667078 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.666909    2577 manager.go:341] "Starting Device Plugin manager"
Nov 03 19:26:12.667174 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.667093    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 19:26:12.667174 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.667123    2577 server.go:87] "Starting device plugin registration server"
Nov 03 19:26:12.667404 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.667393    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 19:26:12.667450 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.667405    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 19:26:12.667538 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.667521    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 19:26:12.667628 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.667616    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 19:26:12.667628 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.667628    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 19:26:12.668187 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.668169    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 19:26:12.668270 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.668222    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-29.us-east-2.compute.internal\" not found"
Nov 03 19:26:12.715551 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.715452    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal"]
Nov 03 19:26:12.715651 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.715558    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:26:12.716561 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.716546    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:26:12.716677 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.716577    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:26:12.716677 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.716589    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:26:12.718701 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.718688    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:26:12.718895 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.718881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:12.718927 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.718913    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:26:12.720728 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.720710    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:26:12.720800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.720730    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:26:12.720800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.720740    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:26:12.720800 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.720753    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:26:12.720912 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.720753    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:26:12.720912 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.720869    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:26:12.722693 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.722679    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:12.722759 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.722703    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:26:12.723404 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.723388    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:26:12.723455 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.723420    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:26:12.723455 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.723433    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:26:12.746031 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.746003    2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-29.us-east-2.compute.internal\" not found" node="ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:12.762490 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.762466    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-29.us-east-2.compute.internal\" not found" node="ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:12.765324 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.765304    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-29.us-east-2.compute.internal\" not found" node="ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:12.768295 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.768282    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:26:12.769154 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.769135    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:26:12.769257 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.769166    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:26:12.769257 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.769179    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:26:12.769257 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.769216    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:12.776505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.776490    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:12.776590 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.776509    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-29.us-east-2.compute.internal\": node \"ip-10-0-133-29.us-east-2.compute.internal\" not found"
Nov 03 19:26:12.789419 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.789402    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-29.us-east-2.compute.internal\" not found"
Nov 03 19:26:12.837510 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.837473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/715d00a389ecabd279054a6ca23fc9ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"715d00a389ecabd279054a6ca23fc9ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:12.837510 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.837507    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd9e417c99cd1e2d8fd47e9d585da591-config\") pod \"kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"dd9e417c99cd1e2d8fd47e9d585da591\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:12.837702 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.837561    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/715d00a389ecabd279054a6ca23fc9ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"715d00a389ecabd279054a6ca23fc9ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:12.889793 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.889762    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-29.us-east-2.compute.internal\" not found"
Nov 03 19:26:12.938173 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.938145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/715d00a389ecabd279054a6ca23fc9ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"715d00a389ecabd279054a6ca23fc9ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:12.938276 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.938183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd9e417c99cd1e2d8fd47e9d585da591-config\") pod \"kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"dd9e417c99cd1e2d8fd47e9d585da591\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:12.938276 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.938219    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd9e417c99cd1e2d8fd47e9d585da591-config\") pod \"kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"dd9e417c99cd1e2d8fd47e9d585da591\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:12.938276 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.938224    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/715d00a389ecabd279054a6ca23fc9ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"715d00a389ecabd279054a6ca23fc9ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:12.938276 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.938250    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/715d00a389ecabd279054a6ca23fc9ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"715d00a389ecabd279054a6ca23fc9ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:12.938276 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:12.938269    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/715d00a389ecabd279054a6ca23fc9ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"715d00a389ecabd279054a6ca23fc9ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:12.990480 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:12.990408    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-29.us-east-2.compute.internal\" not found"
Nov 03 19:26:13.068142 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:13.068117    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:13.070455 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:13.070423    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:13.091140 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:13.091111    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-29.us-east-2.compute.internal\" not found"
Nov 03 19:26:13.191416 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:13.191370    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-29.us-east-2.compute.internal\" not found"
Nov 03 19:26:13.292287 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:13.292256    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-29.us-east-2.compute.internal\" not found"
Nov 03 19:26:13.396676 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:13.396639    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-29.us-east-2.compute.internal\" not found"
Nov 03 19:26:13.459968 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:13.459934    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 19:26:13.460435 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:13.460090    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 19:26:13.499171 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:13.499147    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-29.us-east-2.compute.internal\" not found"
Nov 03 19:26:13.557333 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:13.557240    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:21:12 +0000 UTC" deadline="2028-05-08 02:59:49.419467655 +0000 UTC"
Nov 03 19:26:13.557333 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:13.557273    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13231h33m35.862199564s"
Nov 03 19:26:13.599970 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:13.599926    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-29.us-east-2.compute.internal\" not found"
Nov 03 19:26:13.609378 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:13.609355    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:26:13.701141 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:13.700864    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-29.us-east-2.compute.internal\" not found"
Nov 03 19:26:13.794434 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:13.794392    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:26:13.805134 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:13.805106    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:26:13.832584 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:13.832553    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715d00a389ecabd279054a6ca23fc9ac.slice/crio-ed1e72c10959b9f9352dd0126173841cdfa6acce1eebd375ed73143110000e55 WatchSource:0}: Error finding container ed1e72c10959b9f9352dd0126173841cdfa6acce1eebd375ed73143110000e55: Status 404 returned error can't find the container with id ed1e72c10959b9f9352dd0126173841cdfa6acce1eebd375ed73143110000e55
Nov 03 19:26:13.833092 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:13.833068    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9e417c99cd1e2d8fd47e9d585da591.slice/crio-5c6dedb710a0f80631462172893a46d7295f769277996791e99c4a9614d0998e WatchSource:0}: Error finding container 5c6dedb710a0f80631462172893a46d7295f769277996791e99c4a9614d0998e: Status 404 returned error can't find the container with id 5c6dedb710a0f80631462172893a46d7295f769277996791e99c4a9614d0998e
Nov 03 19:26:13.835256 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:13.835232    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:13.837808 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:13.837793    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:26:13.847845 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:13.847818    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:26:13.849029 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:13.849015    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:26:13.860810 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:13.860789    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:26:14.506783 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.506732    2577 apiserver.go:52] "Watching apiserver"
Nov 03 19:26:14.520681 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.520641    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 19:26:14.521469 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.521440    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-52ft7","kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal","openshift-dns/node-resolver-r2j57","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-s5hhz","openshift-network-operator/iptables-alerter-sp6pz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5","openshift-multus/multus-pc8t6","openshift-network-diagnostics/network-check-target-494mb","openshift-ovn-kubernetes/ovnkube-node-fqmk8","kube-system/global-pull-secret-syncer-lqnwg","openshift-cluster-node-tuning-operator/tuned-6ksh4","openshift-image-registry/node-ca-2n8jc","openshift-multus/network-metrics-daemon-tz6b2"]
Nov 03 19:26:14.524449 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.524418    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.526910 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.526874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:26:14.529609 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.529281    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:26:14.529877 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.529835    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 19:26:14.530327 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.530264    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 19:26:14.530327 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.530275    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 19:26:14.531912 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.531889    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s8pjd\""
Nov 03 19:26:14.532900 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.532214    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:14.532900 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:14.532289    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:14.532900 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.532366    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 19:26:14.532900 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.532412    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 19:26:14.532900 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.532454    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 19:26:14.532900 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.532532    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 19:26:14.532900 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.532797    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 19:26:14.532900 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.532810    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 19:26:14.532900 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.532836    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lmsjd\""
Nov 03 19:26:14.533946 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.533903    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 19:26:14.534614 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.534595    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 19:26:14.534614 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.534611    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xp5ps\""
Nov 03 19:26:14.534874 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.534832    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.535360 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.535344    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 19:26:14.537587 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.537564    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:26:14.540303 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.540279    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:14.540408 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:14.540349    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:14.542707 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.542685    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.545149 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.545133    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.547895 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.547643    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-52ft7"
Nov 03 19:26:14.548647 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.548628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-var-lib-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.548734 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.548656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-node-log\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.548734 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.548678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74bbf640-fdad-4295-8810-f471ee4bb4da-tmp-dir\") pod \"node-resolver-r2j57\" (UID: \"74bbf640-fdad-4295-8810-f471ee4bb4da\") " pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:26:14.548734 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.548694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-slash\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.548734 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.548713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-systemd\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.549037 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.548772    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8hvm\" (UniqueName: \"kubernetes.io/projected/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-kube-api-access-r8hvm\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.549037 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.548808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74bbf640-fdad-4295-8810-f471ee4bb4da-hosts-file\") pod \"node-resolver-r2j57\" (UID: \"74bbf640-fdad-4295-8810-f471ee4bb4da\") " pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:26:14.549037 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.548835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmg4z\" (UniqueName: \"kubernetes.io/projected/74bbf640-fdad-4295-8810-f471ee4bb4da-kube-api-access-dmg4z\") pod \"node-resolver-r2j57\" (UID: \"74bbf640-fdad-4295-8810-f471ee4bb4da\") " pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:26:14.549037 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.548891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdkxd\" (UniqueName: \"kubernetes.io/projected/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-kube-api-access-fdkxd\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.549037 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.548956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-cni-netd\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.549037 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.548983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.549037 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-run-netns\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.549037 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549038    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-log-socket\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.549418 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovnkube-script-lib\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.549418 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/90836408-13df-4995-acec-b7bf40eeec84-serviceca\") pod \"node-ca-2n8jc\" (UID: \"90836408-13df-4995-acec-b7bf40eeec84\") " pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:26:14.549418 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-system-cni-dir\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.549418 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549190    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cnibin\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.549418 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.549418 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-cni-bin\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.549418 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90836408-13df-4995-acec-b7bf40eeec84-host\") pod \"node-ca-2n8jc\" (UID: \"90836408-13df-4995-acec-b7bf40eeec84\") " pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:26:14.549418 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-ovn\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.549418 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.549873 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d79046e-eee4-4336-a852-131f013288a7-iptables-alerter-script\") pod \"iptables-alerter-sp6pz\" (UID: \"7d79046e-eee4-4336-a852-131f013288a7\") " pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:26:14.549873 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-kubelet\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.549873 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-systemd-units\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.549873 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-env-overrides\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.549873 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cni-binary-copy\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.549873 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9fmg\" (UniqueName: \"kubernetes.io/projected/90836408-13df-4995-acec-b7bf40eeec84-kube-api-access-v9fmg\") pod \"node-ca-2n8jc\" (UID: \"90836408-13df-4995-acec-b7bf40eeec84\") " pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:26:14.549873 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549623    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d79046e-eee4-4336-a852-131f013288a7-host-slash\") pod \"iptables-alerter-sp6pz\" (UID: \"7d79046e-eee4-4336-a852-131f013288a7\") " pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:26:14.549873 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b45zq\" (UniqueName: \"kubernetes.io/projected/7d79046e-eee4-4336-a852-131f013288a7-kube-api-access-b45zq\") pod \"iptables-alerter-sp6pz\" (UID: \"7d79046e-eee4-4336-a852-131f013288a7\") " pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:26:14.549873 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.549873 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovnkube-config\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.549873 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovn-node-metrics-cert\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.550445 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.550445 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs\") pod \"network-metrics-daemon-tz6b2\" (UID: \"7750339b-7420-4dda-904e-8221ba6c7ffe\") " pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:14.550445 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.549963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjzq9\" (UniqueName: \"kubernetes.io/projected/7750339b-7420-4dda-904e-8221ba6c7ffe-kube-api-access-mjzq9\") pod \"network-metrics-daemon-tz6b2\" (UID: \"7750339b-7420-4dda-904e-8221ba6c7ffe\") " pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:14.550445 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.550076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.550445 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.550106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-etc-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.550606 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.550448    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:14.550606 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:14.550509    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:14.552893 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.552875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.553034 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.553014    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 19:26:14.574803 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.574776    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qcgcj"
Nov 03 19:26:14.592208 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.592179    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qcgcj"
Nov 03 19:26:14.623000 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.622941    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal" event={"ID":"715d00a389ecabd279054a6ca23fc9ac","Type":"ContainerStarted","Data":"ed1e72c10959b9f9352dd0126173841cdfa6acce1eebd375ed73143110000e55"}
Nov 03 19:26:14.625141 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.624526    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal" event={"ID":"dd9e417c99cd1e2d8fd47e9d585da591","Type":"ContainerStarted","Data":"5c6dedb710a0f80631462172893a46d7295f769277996791e99c4a9614d0998e"}
Nov 03 19:26:14.636943 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.636900    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 19:26:14.639146 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.639123    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:26:14.639441 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.639272    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 19:26:14.639441 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.639375    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 19:26:14.639441 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.639384    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 19:26:14.639441 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.639404    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 19:26:14.639441 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.639423    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 19:26:14.639662 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.639555    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 19:26:14.639662 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.639563    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 19:26:14.639662 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.639605    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 19:26:14.639662 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.639660    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-st58b\""
Nov 03 19:26:14.641986 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.641965    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:26:14.642158 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.642141    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lsdlf\""
Nov 03 19:26:14.642267 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.642244    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f8cx9\""
Nov 03 19:26:14.642376 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.642326    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 19:26:14.642376 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.642368    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 19:26:14.642475 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.642441    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-hwgf7\""
Nov 03 19:26:14.642520 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.642477    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 19:26:14.642556 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.642528    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g6kk8\""
Nov 03 19:26:14.642601 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.642555    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 19:26:14.642601 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.642597    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 19:26:14.642682 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.642639    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r6b5k\""
Nov 03 19:26:14.642725 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.642714    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 19:26:14.651162 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.651310 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.651310 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-socket-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.651310 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-netns\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.651310 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651245    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-tmp\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.651310 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651260    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-k8s-cni-cncf-io\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.651310 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651282    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-cni-bin\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.651310 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-systemd\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.651669 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651280    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.651669 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651381    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-systemd\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.651669 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvttl\" (UniqueName: \"kubernetes.io/projected/f0efd5bb-5c99-4c84-a77f-d816995e6f79-kube-api-access-cvttl\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.651669 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-kubelet\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.651669 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-run\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.651669 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-lib-modules\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.651669 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651505    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl729\" (UniqueName: \"kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729\") pod \"network-check-target-494mb\" (UID: \"aef23c0d-d927-4a6e-bc22-288027b498a8\") " pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:14.651669 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmg4z\" (UniqueName: \"kubernetes.io/projected/74bbf640-fdad-4295-8810-f471ee4bb4da-kube-api-access-dmg4z\") pod \"node-resolver-r2j57\" (UID: \"74bbf640-fdad-4295-8810-f471ee4bb4da\") " pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:26:14.651669 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651527    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.651669 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdkxd\" (UniqueName: \"kubernetes.io/projected/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-kube-api-access-fdkxd\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.651669 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651596    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-cni-netd\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.651669 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651615    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.651669 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-systemd\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.651669 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-run-netns\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.651669 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-log-socket\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.652309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-conf-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.652309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651740    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysctl-d\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.652309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651757    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysctl-conf\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.652309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-system-cni-dir\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.652309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cnibin\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.652309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90836408-13df-4995-acec-b7bf40eeec84-host\") pod \"node-ca-2n8jc\" (UID: \"90836408-13df-4995-acec-b7bf40eeec84\") " pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:26:14.652309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651851    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-run-netns\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.652309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-ovn\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.652309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651927    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-ovn\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.652309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651929    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90836408-13df-4995-acec-b7bf40eeec84-host\") pod \"node-ca-2n8jc\" (UID: \"90836408-13df-4995-acec-b7bf40eeec84\") " pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:26:14.652309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651954    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-log-socket\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.652309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651976    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cnibin\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.652309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.651988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-kubelet\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.652309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-systemd-units\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.652309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-env-overrides\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.652309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-device-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.652309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-cnibin\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.652964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-tuned\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.652964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652172    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.652964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cni-binary-copy\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.652964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652205    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-kubelet\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.652964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9fmg\" (UniqueName: \"kubernetes.io/projected/90836408-13df-4995-acec-b7bf40eeec84-kube-api-access-v9fmg\") pod \"node-ca-2n8jc\" (UID: \"90836408-13df-4995-acec-b7bf40eeec84\") " pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:26:14.652964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652238    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-cni-netd\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.652964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652247    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-cni-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.652964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652385    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-system-cni-dir\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.652964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652438    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grpgt\" (UniqueName: \"kubernetes.io/projected/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-kube-api-access-grpgt\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.652964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9095ae71-1326-4ec3-9755-164738a4da17-agent-certs\") pod \"konnectivity-agent-52ft7\" (UID: \"9095ae71-1326-4ec3-9755-164738a4da17\") " pod="kube-system/konnectivity-agent-52ft7"
Nov 03 19:26:14.652964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d79046e-eee4-4336-a852-131f013288a7-host-slash\") pod \"iptables-alerter-sp6pz\" (UID: \"7d79046e-eee4-4336-a852-131f013288a7\") " pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:26:14.652964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovnkube-config\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.652964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovn-node-metrics-cert\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.652964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs\") pod \"network-metrics-daemon-tz6b2\" (UID: \"7750339b-7420-4dda-904e-8221ba6c7ffe\") " pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:14.652964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjzq9\" (UniqueName: \"kubernetes.io/projected/7750339b-7420-4dda-904e-8221ba6c7ffe-kube-api-access-mjzq9\") pod \"network-metrics-daemon-tz6b2\" (UID: \"7750339b-7420-4dda-904e-8221ba6c7ffe\") " pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:14.652964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-etc-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.652964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-cni-multus\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.653660 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-kubernetes\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.653660 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9095ae71-1326-4ec3-9755-164738a4da17-konnectivity-ca\") pod \"konnectivity-agent-52ft7\" (UID: \"9095ae71-1326-4ec3-9755-164738a4da17\") " pod="kube-system/konnectivity-agent-52ft7"
Nov 03 19:26:14.653660 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:14.653660 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-var-lib-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.653660 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-node-log\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.653660 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652864    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-systemd-units\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.653660 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652880    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cni-binary-copy\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.653660 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652940    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-var-lib-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.653660 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652825    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-env-overrides\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.653660 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.652968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74bbf640-fdad-4295-8810-f471ee4bb4da-tmp-dir\") pod \"node-resolver-r2j57\" (UID: \"74bbf640-fdad-4295-8810-f471ee4bb4da\") " pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:26:14.653660 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:14.652987    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:26:14.653660 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-slash\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.653660 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8hvm\" (UniqueName: \"kubernetes.io/projected/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-kube-api-access-r8hvm\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.653660 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:14.653059    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs podName:7750339b-7420-4dda-904e-8221ba6c7ffe nodeName:}" failed. No retries permitted until 2026-11-03 19:26:15.153032055 +0000 UTC m=+3.002246133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs") pod "network-metrics-daemon-tz6b2" (UID: "7750339b-7420-4dda-904e-8221ba6c7ffe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:26:14.653660 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653287    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 19:26:14.653660 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653328    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.653660 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653381    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74bbf640-fdad-4295-8810-f471ee4bb4da-hosts-file\") pod \"node-resolver-r2j57\" (UID: \"74bbf640-fdad-4295-8810-f471ee4bb4da\") " pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:26:14.654505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653400    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-node-log\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.654505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-etc-selinux\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.654505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653452    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d79046e-eee4-4336-a852-131f013288a7-host-slash\") pod \"iptables-alerter-sp6pz\" (UID: \"7d79046e-eee4-4336-a852-131f013288a7\") " pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:26:14.654505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653468    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-sys-fs\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.654505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-socket-dir-parent\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.654505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-etc-kubernetes\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.654505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-host\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.654505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovnkube-script-lib\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.654505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/90836408-13df-4995-acec-b7bf40eeec84-serviceca\") pod \"node-ca-2n8jc\" (UID: \"90836408-13df-4995-acec-b7bf40eeec84\") " pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:26:14.654505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653892    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-etc-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.654505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653903    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovnkube-config\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.654505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-registration-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.654505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.653968    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-slash\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.654505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.654017    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74bbf640-fdad-4295-8810-f471ee4bb4da-hosts-file\") pod \"node-resolver-r2j57\" (UID: \"74bbf640-fdad-4295-8810-f471ee4bb4da\") " pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:26:14.654505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.654255    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74bbf640-fdad-4295-8810-f471ee4bb4da-tmp-dir\") pod \"node-resolver-r2j57\" (UID: \"74bbf640-fdad-4295-8810-f471ee4bb4da\") " pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:26:14.654505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.654275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.654505 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.654337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-cni-bin\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.655399 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.654416    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-cni-bin\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.655399 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.654418    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-daemon-config\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.655399 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.654501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-var-lib-kubelet\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.655399 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.654580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b22e16a-511b-4d70-a39c-16160ef790b1-dbus\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:14.655399 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.654644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.655399 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.654693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d79046e-eee4-4336-a852-131f013288a7-iptables-alerter-script\") pod \"iptables-alerter-sp6pz\" (UID: \"7d79046e-eee4-4336-a852-131f013288a7\") " pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:26:14.655399 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.654738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np75z\" (UniqueName: \"kubernetes.io/projected/98108a97-4bce-4291-8c25-ef6a88d542e4-kube-api-access-np75z\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.655399 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.654814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-modprobe-d\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.655399 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.654841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-sys\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.655399 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.654886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98108a97-4bce-4291-8c25-ef6a88d542e4-cni-binary-copy\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.655399 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.654911    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-hostroot\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.655399 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.654938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-multus-certs\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.655399 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.654968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b45zq\" (UniqueName: \"kubernetes.io/projected/7d79046e-eee4-4336-a852-131f013288a7-kube-api-access-b45zq\") pod \"iptables-alerter-sp6pz\" (UID: \"7d79046e-eee4-4336-a852-131f013288a7\") " pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:26:14.655399 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.655044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.655399 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.655063    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovnkube-script-lib\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.655399 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.655074    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-system-cni-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.655399 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.655104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysconfig\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.656121 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.655131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b22e16a-511b-4d70-a39c-16160ef790b1-kubelet-config\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:14.656121 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.655248    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.656121 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.655567    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d79046e-eee4-4336-a852-131f013288a7-iptables-alerter-script\") pod \"iptables-alerter-sp6pz\" (UID: \"7d79046e-eee4-4336-a852-131f013288a7\") " pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:26:14.656121 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.655799    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/90836408-13df-4995-acec-b7bf40eeec84-serviceca\") pod \"node-ca-2n8jc\" (UID: \"90836408-13df-4995-acec-b7bf40eeec84\") " pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:26:14.656261 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.656122    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.656624 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.656603    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.658409 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.658381    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovn-node-metrics-cert\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.665704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.665676    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmg4z\" (UniqueName: \"kubernetes.io/projected/74bbf640-fdad-4295-8810-f471ee4bb4da-kube-api-access-dmg4z\") pod \"node-resolver-r2j57\" (UID: \"74bbf640-fdad-4295-8810-f471ee4bb4da\") " pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:26:14.665847 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.665811    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdkxd\" (UniqueName: \"kubernetes.io/projected/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-kube-api-access-fdkxd\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.673283 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.673253    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9fmg\" (UniqueName: \"kubernetes.io/projected/90836408-13df-4995-acec-b7bf40eeec84-kube-api-access-v9fmg\") pod \"node-ca-2n8jc\" (UID: \"90836408-13df-4995-acec-b7bf40eeec84\") " pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:26:14.673283 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.673262    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8hvm\" (UniqueName: \"kubernetes.io/projected/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-kube-api-access-r8hvm\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.674123 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.674085    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b45zq\" (UniqueName: \"kubernetes.io/projected/7d79046e-eee4-4336-a852-131f013288a7-kube-api-access-b45zq\") pod \"iptables-alerter-sp6pz\" (UID: \"7d79046e-eee4-4336-a852-131f013288a7\") " pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:26:14.674251 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.674136    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjzq9\" (UniqueName: \"kubernetes.io/projected/7750339b-7420-4dda-904e-8221ba6c7ffe-kube-api-access-mjzq9\") pod \"network-metrics-daemon-tz6b2\" (UID: \"7750339b-7420-4dda-904e-8221ba6c7ffe\") " pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:14.756419 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-device-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.756596 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-cnibin\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.756596 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-tuned\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.756596 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-cni-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.756596 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grpgt\" (UniqueName: \"kubernetes.io/projected/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-kube-api-access-grpgt\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.756596 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9095ae71-1326-4ec3-9755-164738a4da17-agent-certs\") pod \"konnectivity-agent-52ft7\" (UID: \"9095ae71-1326-4ec3-9755-164738a4da17\") " pod="kube-system/konnectivity-agent-52ft7"
Nov 03 19:26:14.756596 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756545    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-device-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.756984 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-cni-multus\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.756984 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-kubernetes\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.756984 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9095ae71-1326-4ec3-9755-164738a4da17-konnectivity-ca\") pod \"konnectivity-agent-52ft7\" (UID: \"9095ae71-1326-4ec3-9755-164738a4da17\") " pod="kube-system/konnectivity-agent-52ft7"
Nov 03 19:26:14.756984 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:14.756984 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756722    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.756984 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-etc-selinux\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.756984 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-sys-fs\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.756984 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-socket-dir-parent\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.756984 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-etc-kubernetes\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.756984 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-host\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.756984 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-registration-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.756984 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-daemon-config\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.756984 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-var-lib-kubelet\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.756984 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.756983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b22e16a-511b-4d70-a39c-16160ef790b1-dbus\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np75z\" (UniqueName: \"kubernetes.io/projected/98108a97-4bce-4291-8c25-ef6a88d542e4-kube-api-access-np75z\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757074    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-sys-fs\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-modprobe-d\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-sys\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98108a97-4bce-4291-8c25-ef6a88d542e4-cni-binary-copy\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-hostroot\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-multus-certs\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-system-cni-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysconfig\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b22e16a-511b-4d70-a39c-16160ef790b1-kubelet-config\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-socket-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-netns\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-tmp\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-k8s-cni-cncf-io\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-cni-bin\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvttl\" (UniqueName: \"kubernetes.io/projected/f0efd5bb-5c99-4c84-a77f-d816995e6f79-kube-api-access-cvttl\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757400    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-socket-dir-parent\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.757704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-kubelet\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.758593 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-run\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.758593 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-lib-modules\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.758593 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757466    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-host\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.758593 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl729\" (UniqueName: \"kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729\") pod \"network-check-target-494mb\" (UID: \"aef23c0d-d927-4a6e-bc22-288027b498a8\") " pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:14.758593 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-systemd\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.758593 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-conf-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.758593 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757518    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-registration-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.758593 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysctl-d\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.758593 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysctl-conf\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.758593 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757587    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-var-lib-kubelet\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.758593 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757642    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-kubernetes\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.758593 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757720    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b22e16a-511b-4d70-a39c-16160ef790b1-dbus\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:14.758593 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757792    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-netns\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.758593 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757844    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysctl-conf\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.758593 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:14.757922    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:26:14.758593 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757963    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-cnibin\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.758593 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:14.757993    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret podName:7b22e16a-511b-4d70-a39c-16160ef790b1 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:15.257970161 +0000 UTC m=+3.107184236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret") pod "global-pull-secret-syncer-lqnwg" (UID: "7b22e16a-511b-4d70-a39c-16160ef790b1") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758024    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-run\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758048    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758101    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-k8s-cni-cncf-io\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758132    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9095ae71-1326-4ec3-9755-164738a4da17-konnectivity-ca\") pod \"konnectivity-agent-52ft7\" (UID: \"9095ae71-1326-4ec3-9755-164738a4da17\") " pod="kube-system/konnectivity-agent-52ft7"
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758150    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-cni-bin\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758204    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-kubelet\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758243    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-etc-kubernetes\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758268    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-lib-modules\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758301    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-cni-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758343    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-multus-certs\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758450    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-etc-selinux\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758529    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-modprobe-d\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758578    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-sys\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758741    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-system-cni-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758781    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-systemd\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758806    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysconfig\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758839    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-conf-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.759447 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758882    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b22e16a-511b-4d70-a39c-16160ef790b1-kubelet-config\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:14.760344 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.758959    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysctl-d\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.760344 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.759007    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-socket-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.760344 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.757886    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-cni-multus\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.760344 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.759129    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-hostroot\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.760344 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.760141    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-daemon-config\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.760344 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.760283    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-tuned\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.760634 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.760491    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98108a97-4bce-4291-8c25-ef6a88d542e4-cni-binary-copy\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.760755 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.760733    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-tmp\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.761730 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.761709    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9095ae71-1326-4ec3-9755-164738a4da17-agent-certs\") pod \"konnectivity-agent-52ft7\" (UID: \"9095ae71-1326-4ec3-9755-164738a4da17\") " pod="kube-system/konnectivity-agent-52ft7"
Nov 03 19:26:14.772797 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:14.772772    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:26:14.772797 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:14.772801    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:26:14.772797 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:14.772816    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hl729 for pod openshift-network-diagnostics/network-check-target-494mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:26:14.773130 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:14.772908    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729 podName:aef23c0d-d927-4a6e-bc22-288027b498a8 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:15.2728852 +0000 UTC m=+3.122099277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hl729" (UniqueName: "kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729") pod "network-check-target-494mb" (UID: "aef23c0d-d927-4a6e-bc22-288027b498a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:26:14.775675 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.775645    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np75z\" (UniqueName: \"kubernetes.io/projected/98108a97-4bce-4291-8c25-ef6a88d542e4-kube-api-access-np75z\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.775782 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.775747    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvttl\" (UniqueName: \"kubernetes.io/projected/f0efd5bb-5c99-4c84-a77f-d816995e6f79-kube-api-access-cvttl\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.776964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.776944    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grpgt\" (UniqueName: \"kubernetes.io/projected/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-kube-api-access-grpgt\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.857169 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.857128    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:26:14.859325 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.859296    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:26:14.864115 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.864084    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:14.873715 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:14.873687    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90836408_13df_4995_acec_b7bf40eeec84.slice/crio-b66f962f7121d724fd502890519de74e9dbbf3005decd9e714b0a1c4bf68235d WatchSource:0}: Error finding container b66f962f7121d724fd502890519de74e9dbbf3005decd9e714b0a1c4bf68235d: Status 404 returned error can't find the container with id b66f962f7121d724fd502890519de74e9dbbf3005decd9e714b0a1c4bf68235d
Nov 03 19:26:14.874728 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:14.874700    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bbf640_fdad_4295_8810_f471ee4bb4da.slice/crio-ba441394447e92ee1a6f4075a70e010b40e3b0d1f12a3347950676421f6d7491 WatchSource:0}: Error finding container ba441394447e92ee1a6f4075a70e010b40e3b0d1f12a3347950676421f6d7491: Status 404 returned error can't find the container with id ba441394447e92ee1a6f4075a70e010b40e3b0d1f12a3347950676421f6d7491
Nov 03 19:26:14.879406 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:14.879381    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cda745_c0e3_45ce_bcb0_5e99dea4f1e4.slice/crio-92ec458682e0252331cafbcd2ee616121af1ba6cd2a5e758a832c08908d25d10 WatchSource:0}: Error finding container 92ec458682e0252331cafbcd2ee616121af1ba6cd2a5e758a832c08908d25d10: Status 404 returned error can't find the container with id 92ec458682e0252331cafbcd2ee616121af1ba6cd2a5e758a832c08908d25d10
Nov 03 19:26:14.882645 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.882623    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:26:14.884898 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.884879    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:26:14.893047 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.893018    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:26:14.894377 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:14.894357    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8251a8_f48e_4af4_9606_5cbdea4a6b96.slice/crio-8c116c6528d4b874734bde912eceb384ec550b0f7ecdc313fa3914e82e2d1db0 WatchSource:0}: Error finding container 8c116c6528d4b874734bde912eceb384ec550b0f7ecdc313fa3914e82e2d1db0: Status 404 returned error can't find the container with id 8c116c6528d4b874734bde912eceb384ec550b0f7ecdc313fa3914e82e2d1db0
Nov 03 19:26:14.894996 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:14.894974    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d79046e_eee4_4336_a852_131f013288a7.slice/crio-98d32d8c2eb7304220d335c426e8ff22f09c940d9c5dd8aab874195c26b8df97 WatchSource:0}: Error finding container 98d32d8c2eb7304220d335c426e8ff22f09c940d9c5dd8aab874195c26b8df97: Status 404 returned error can't find the container with id 98d32d8c2eb7304220d335c426e8ff22f09c940d9c5dd8aab874195c26b8df97
Nov 03 19:26:14.895827 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.895802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pc8t6"
Nov 03 19:26:14.905979 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.905957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-52ft7"
Nov 03 19:26:14.906266 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:14.906237    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0efd5bb_5c99_4c84_a77f_d816995e6f79.slice/crio-396d324ef98b37e05f1034cfc35e813cc772e3cfd1065b4dbe33b1d2a2e71c9a WatchSource:0}: Error finding container 396d324ef98b37e05f1034cfc35e813cc772e3cfd1065b4dbe33b1d2a2e71c9a: Status 404 returned error can't find the container with id 396d324ef98b37e05f1034cfc35e813cc772e3cfd1065b4dbe33b1d2a2e71c9a
Nov 03 19:26:14.907110 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:14.907006    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98108a97_4bce_4291_8c25_ef6a88d542e4.slice/crio-0b7f51a479baf646f10794133fc19ecf46c03507fcd900f3f6ec74527c0069fb WatchSource:0}: Error finding container 0b7f51a479baf646f10794133fc19ecf46c03507fcd900f3f6ec74527c0069fb: Status 404 returned error can't find the container with id 0b7f51a479baf646f10794133fc19ecf46c03507fcd900f3f6ec74527c0069fb
Nov 03 19:26:14.912141 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.912114    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:26:14.918676 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:14.918642    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9095ae71_1326_4ec3_9755_164738a4da17.slice/crio-cb65a1d8f94b24f8ced23ba2c44cc83e756d780b4b80025e2022ceba665fc1bd WatchSource:0}: Error finding container cb65a1d8f94b24f8ced23ba2c44cc83e756d780b4b80025e2022ceba665fc1bd: Status 404 returned error can't find the container with id cb65a1d8f94b24f8ced23ba2c44cc83e756d780b4b80025e2022ceba665fc1bd
Nov 03 19:26:14.921301 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:14.921276    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:26:14.922777 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:26:14.922743    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8273a1a4_5dbb_4f54_8f4c_4207cc11c228.slice/crio-92007d598bf97aa055fdb49040299a302f2482536d3e301ad06abdfe4dc77a02 WatchSource:0}: Error finding container 92007d598bf97aa055fdb49040299a302f2482536d3e301ad06abdfe4dc77a02: Status 404 returned error can't find the container with id 92007d598bf97aa055fdb49040299a302f2482536d3e301ad06abdfe4dc77a02
Nov 03 19:26:15.161341 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:15.161300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs\") pod \"network-metrics-daemon-tz6b2\" (UID: \"7750339b-7420-4dda-904e-8221ba6c7ffe\") " pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:15.161559 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:15.161483    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:26:15.161626 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:15.161574    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs podName:7750339b-7420-4dda-904e-8221ba6c7ffe nodeName:}" failed. No retries permitted until 2026-11-03 19:26:16.161551148 +0000 UTC m=+4.010765244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs") pod "network-metrics-daemon-tz6b2" (UID: "7750339b-7420-4dda-904e-8221ba6c7ffe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:26:15.262378 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:15.262342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:15.262559 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:15.262490    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:26:15.262639 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:15.262575    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret podName:7b22e16a-511b-4d70-a39c-16160ef790b1 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:16.262544634 +0000 UTC m=+4.111758714 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret") pod "global-pull-secret-syncer-lqnwg" (UID: "7b22e16a-511b-4d70-a39c-16160ef790b1") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:26:15.363592 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:15.363459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl729\" (UniqueName: \"kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729\") pod \"network-check-target-494mb\" (UID: \"aef23c0d-d927-4a6e-bc22-288027b498a8\") " pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:15.363795 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:15.363652    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:26:15.363795 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:15.363675    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:26:15.363795 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:15.363688    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hl729 for pod openshift-network-diagnostics/network-check-target-494mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:26:15.363795 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:15.363742    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729 podName:aef23c0d-d927-4a6e-bc22-288027b498a8 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:16.363726806 +0000 UTC m=+4.212940870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hl729" (UniqueName: "kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729") pod "network-check-target-494mb" (UID: "aef23c0d-d927-4a6e-bc22-288027b498a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:26:15.595310 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:15.595173    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:21:14 +0000 UTC" deadline="2028-06-18 07:32:26.698526709 +0000 UTC"
Nov 03 19:26:15.595310 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:15.595239    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14220h6m11.103291927s"
Nov 03 19:26:15.615871 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:15.615326    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:15.615871 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:15.615471    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:15.637130 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:15.637032    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6ksh4" event={"ID":"8273a1a4-5dbb-4f54-8f4c-4207cc11c228","Type":"ContainerStarted","Data":"92007d598bf97aa055fdb49040299a302f2482536d3e301ad06abdfe4dc77a02"}
Nov 03 19:26:15.655745 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:15.655594    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pc8t6" event={"ID":"98108a97-4bce-4291-8c25-ef6a88d542e4","Type":"ContainerStarted","Data":"0b7f51a479baf646f10794133fc19ecf46c03507fcd900f3f6ec74527c0069fb"}
Nov 03 19:26:15.666907 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:15.666807    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5hhz" event={"ID":"cb8251a8-f48e-4af4-9606-5cbdea4a6b96","Type":"ContainerStarted","Data":"8c116c6528d4b874734bde912eceb384ec550b0f7ecdc313fa3914e82e2d1db0"}
Nov 03 19:26:15.672042 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:15.671901    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8" event={"ID":"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4","Type":"ContainerStarted","Data":"92ec458682e0252331cafbcd2ee616121af1ba6cd2a5e758a832c08908d25d10"}
Nov 03 19:26:15.676499 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:15.676408    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2n8jc" event={"ID":"90836408-13df-4995-acec-b7bf40eeec84","Type":"ContainerStarted","Data":"b66f962f7121d724fd502890519de74e9dbbf3005decd9e714b0a1c4bf68235d"}
Nov 03 19:26:15.680978 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:15.680939    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-52ft7" event={"ID":"9095ae71-1326-4ec3-9755-164738a4da17","Type":"ContainerStarted","Data":"cb65a1d8f94b24f8ced23ba2c44cc83e756d780b4b80025e2022ceba665fc1bd"}
Nov 03 19:26:15.692884 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:15.692805    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5" event={"ID":"f0efd5bb-5c99-4c84-a77f-d816995e6f79","Type":"ContainerStarted","Data":"396d324ef98b37e05f1034cfc35e813cc772e3cfd1065b4dbe33b1d2a2e71c9a"}
Nov 03 19:26:15.708261 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:15.708215    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sp6pz" event={"ID":"7d79046e-eee4-4336-a852-131f013288a7","Type":"ContainerStarted","Data":"98d32d8c2eb7304220d335c426e8ff22f09c940d9c5dd8aab874195c26b8df97"}
Nov 03 19:26:15.714693 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:15.714637    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r2j57" event={"ID":"74bbf640-fdad-4295-8810-f471ee4bb4da","Type":"ContainerStarted","Data":"ba441394447e92ee1a6f4075a70e010b40e3b0d1f12a3347950676421f6d7491"}
Nov 03 19:26:16.170789 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:16.170125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs\") pod \"network-metrics-daemon-tz6b2\" (UID: \"7750339b-7420-4dda-904e-8221ba6c7ffe\") " pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:16.170789 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:16.170325    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:26:16.170789 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:16.170397    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs podName:7750339b-7420-4dda-904e-8221ba6c7ffe nodeName:}" failed. No retries permitted until 2026-11-03 19:26:18.170378852 +0000 UTC m=+6.019592938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs") pod "network-metrics-daemon-tz6b2" (UID: "7750339b-7420-4dda-904e-8221ba6c7ffe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:26:16.271491 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:16.271446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:16.271673 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:16.271638    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:26:16.271747 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:16.271697    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret podName:7b22e16a-511b-4d70-a39c-16160ef790b1 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:18.271682825 +0000 UTC m=+6.120896897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret") pod "global-pull-secret-syncer-lqnwg" (UID: "7b22e16a-511b-4d70-a39c-16160ef790b1") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:26:16.372080 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:16.372037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl729\" (UniqueName: \"kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729\") pod \"network-check-target-494mb\" (UID: \"aef23c0d-d927-4a6e-bc22-288027b498a8\") " pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:16.372314 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:16.372294    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:26:16.372378 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:16.372324    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:26:16.372378 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:16.372340    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hl729 for pod openshift-network-diagnostics/network-check-target-494mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:26:16.372500 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:16.372405    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729 podName:aef23c0d-d927-4a6e-bc22-288027b498a8 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:18.372386044 +0000 UTC m=+6.221600122 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hl729" (UniqueName: "kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729") pod "network-check-target-494mb" (UID: "aef23c0d-d927-4a6e-bc22-288027b498a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:26:16.595551 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:16.595472    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:21:14 +0000 UTC" deadline="2028-07-24 16:24:28.91826728 +0000 UTC"
Nov 03 19:26:16.595551 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:16.595547    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15092h58m12.32272458s"
Nov 03 19:26:16.617985 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:16.617219    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:16.617985 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:16.617359    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:16.617985 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:16.617784    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:16.617985 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:16.617893    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:17.615633 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:17.615390    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:17.616073 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:17.615702    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:17.758892 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:17.744537    2577 generic.go:356] "Generic (PLEG): container finished" podID="715d00a389ecabd279054a6ca23fc9ac" containerID="d0c5364d8ec8d2c3737218feb52caa79ff3e361cb6effbde7328099d809eacc2" exitCode=0
Nov 03 19:26:17.758892 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:17.744651    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal" event={"ID":"715d00a389ecabd279054a6ca23fc9ac","Type":"ContainerDied","Data":"d0c5364d8ec8d2c3737218feb52caa79ff3e361cb6effbde7328099d809eacc2"}
Nov 03 19:26:17.758892 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:17.747470    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal" event={"ID":"dd9e417c99cd1e2d8fd47e9d585da591","Type":"ContainerStarted","Data":"244fe5d5b0655def75d29c33708816f58eb1936c6bc88ea714e689eb2cd893d0"}
Nov 03 19:26:18.195139 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:18.195056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs\") pod \"network-metrics-daemon-tz6b2\" (UID: \"7750339b-7420-4dda-904e-8221ba6c7ffe\") " pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:18.195316 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:18.195286    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:26:18.195392 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:18.195364    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs podName:7750339b-7420-4dda-904e-8221ba6c7ffe nodeName:}" failed. No retries permitted until 2026-11-03 19:26:22.195344898 +0000 UTC m=+10.044558972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs") pod "network-metrics-daemon-tz6b2" (UID: "7750339b-7420-4dda-904e-8221ba6c7ffe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:26:18.295972 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:18.295918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:18.296128 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:18.296094    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:26:18.296213 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:18.296165    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret podName:7b22e16a-511b-4d70-a39c-16160ef790b1 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:22.296145178 +0000 UTC m=+10.145359260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret") pod "global-pull-secret-syncer-lqnwg" (UID: "7b22e16a-511b-4d70-a39c-16160ef790b1") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:26:18.398716 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:18.397244    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl729\" (UniqueName: \"kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729\") pod \"network-check-target-494mb\" (UID: \"aef23c0d-d927-4a6e-bc22-288027b498a8\") " pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:18.398716 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:18.397473    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:26:18.398716 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:18.397534    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:26:18.398716 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:18.397550    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hl729 for pod openshift-network-diagnostics/network-check-target-494mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:26:18.398716 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:18.397617    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729 podName:aef23c0d-d927-4a6e-bc22-288027b498a8 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:22.397598494 +0000 UTC m=+10.246812570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hl729" (UniqueName: "kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729") pod "network-check-target-494mb" (UID: "aef23c0d-d927-4a6e-bc22-288027b498a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:26:18.617636 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:18.617593    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:18.618148 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:18.617744    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:18.618148 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:18.618126    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:18.618274 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:18.618207    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:18.756038 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:18.754930    2577 generic.go:356] "Generic (PLEG): container finished" podID="cb8251a8-f48e-4af4-9606-5cbdea4a6b96" containerID="76c4fa47a463ab9241093ceb9f604556117e574983b4f18ccc5de032bc44936d" exitCode=0
Nov 03 19:26:18.756038 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:18.755052    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5hhz" event={"ID":"cb8251a8-f48e-4af4-9606-5cbdea4a6b96","Type":"ContainerDied","Data":"76c4fa47a463ab9241093ceb9f604556117e574983b4f18ccc5de032bc44936d"}
Nov 03 19:26:18.772154 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:18.772034    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal" event={"ID":"715d00a389ecabd279054a6ca23fc9ac","Type":"ContainerStarted","Data":"9541278ef73e638d1920a6e8ba31e6ddaf76d01439d9f6b27d1263b8b0e1827a"}
Nov 03 19:26:18.793035 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:18.791343    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal" podStartSLOduration=4.791319598 podStartE2EDuration="4.791319598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:17.78762031 +0000 UTC m=+5.636834395" watchObservedRunningTime="2026-11-03 19:26:18.791319598 +0000 UTC m=+6.640533683"
Nov 03 19:26:19.615765 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:19.615715    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:19.615953 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:19.615852    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:19.778934 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:19.778899    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-52ft7" event={"ID":"9095ae71-1326-4ec3-9755-164738a4da17","Type":"ContainerStarted","Data":"25c453a1ce4eb8abfe484d5a964ded5f02eef3cde13770dfb573fb065329ee61"}
Nov 03 19:26:19.797949 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:19.797890    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-52ft7" podStartSLOduration=2.448182153 podStartE2EDuration="6.797872163s" totalImagesPullingTime="4.34969001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:19.797782119 +0000 UTC m=+7.646996210" watchObservedRunningTime="2026-11-03 19:26:19.797872163 +0000 UTC m=+7.647086243"
Nov 03 19:26:19.798143 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:19.798093    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal" podStartSLOduration=5.798082525 podStartE2EDuration="5.798082525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:18.813196363 +0000 UTC m=+6.662410446" watchObservedRunningTime="2026-11-03 19:26:19.798082525 +0000 UTC m=+7.647296604"
Nov 03 19:26:20.616929 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:20.616898    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:20.617079 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:20.617026    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:20.617394 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:20.617364    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:20.617493 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:20.617459    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:20.787046 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:20.787013    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2n8jc" event={"ID":"90836408-13df-4995-acec-b7bf40eeec84","Type":"ContainerStarted","Data":"2246a4ef3959771547e6c2b7775b9de0ee9ec58e22087133cb4bb5cff9b85f0f"}
Nov 03 19:26:20.812322 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:20.812258    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2n8jc" podStartSLOduration=2.270107978 podStartE2EDuration="7.812241304s" totalImagesPullingTime="5.542133326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:20.811411495 +0000 UTC m=+8.660625576" watchObservedRunningTime="2026-11-03 19:26:20.812241304 +0000 UTC m=+8.661455386"
Nov 03 19:26:20.859656 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:20.859618    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-52ft7"
Nov 03 19:26:20.860136 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:20.860112    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-52ft7"
Nov 03 19:26:21.614910 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:21.614875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:21.615098 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:21.615016    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:21.791200 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:21.791164    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5" event={"ID":"f0efd5bb-5c99-4c84-a77f-d816995e6f79","Type":"ContainerStarted","Data":"d2d9626bf1df53004bf8e61c12032866d14ec859758f1bee18eeae9fb8ac0fa4"}
Nov 03 19:26:22.230850 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:22.230802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs\") pod \"network-metrics-daemon-tz6b2\" (UID: \"7750339b-7420-4dda-904e-8221ba6c7ffe\") " pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:22.231016 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:22.230956    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:26:22.231016 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:22.231005    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs podName:7750339b-7420-4dda-904e-8221ba6c7ffe nodeName:}" failed. No retries permitted until 2026-11-03 19:26:30.230991216 +0000 UTC m=+18.080205273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs") pod "network-metrics-daemon-tz6b2" (UID: "7750339b-7420-4dda-904e-8221ba6c7ffe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:26:22.331436 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:22.331404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:22.331592 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:22.331540    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:26:22.331592 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:22.331587    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret podName:7b22e16a-511b-4d70-a39c-16160ef790b1 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:30.331574438 +0000 UTC m=+18.180788495 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret") pod "global-pull-secret-syncer-lqnwg" (UID: "7b22e16a-511b-4d70-a39c-16160ef790b1") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:26:22.432459 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:22.432414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl729\" (UniqueName: \"kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729\") pod \"network-check-target-494mb\" (UID: \"aef23c0d-d927-4a6e-bc22-288027b498a8\") " pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:22.432621 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:22.432586    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:26:22.432621 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:22.432607    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:26:22.432621 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:22.432622    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hl729 for pod openshift-network-diagnostics/network-check-target-494mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:26:22.432798 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:22.432683    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729 podName:aef23c0d-d927-4a6e-bc22-288027b498a8 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:30.432664694 +0000 UTC m=+18.281878764 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hl729" (UniqueName: "kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729") pod "network-check-target-494mb" (UID: "aef23c0d-d927-4a6e-bc22-288027b498a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:26:22.617012 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:22.616969    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:22.617194 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:22.617108    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:22.618407 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:22.617327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:22.618407 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:22.617430    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:22.790607 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:22.790575    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 19:26:22.811373 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:22.811345    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:26:22.811893 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:22.811868    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5" event={"ID":"f0efd5bb-5c99-4c84-a77f-d816995e6f79","Type":"ContainerStarted","Data":"06223fa00cad34fa6ae2e7ae81ae8be2500bbf0ebd296c52495546b7b8328c67"}
Nov 03 19:26:23.615106 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:23.615077    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:23.615236 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:23.615215    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:23.675508 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:23.675406    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:26:22.790602003Z","UUID":"bce22a51-6bd8-43b9-bf36-cd6cf9986a71","Handler":null,"Name":"","Endpoint":""}
Nov 03 19:26:23.677209 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:23.676979    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 19:26:23.677315 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:23.677221    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 19:26:23.815739 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:23.815699    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r2j57" event={"ID":"74bbf640-fdad-4295-8810-f471ee4bb4da","Type":"ContainerStarted","Data":"f79e38385733769744d9cf86373e608824041bfc599ae2244beea8d989f734e0"}
Nov 03 19:26:23.818133 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:23.818100    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pc8t6" event={"ID":"98108a97-4bce-4291-8c25-ef6a88d542e4","Type":"ContainerStarted","Data":"aa6383c3b90c6154f2344381939d93b937510d1feef7cb5c6256b5810f72478e"}
Nov 03 19:26:23.820999 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:23.820976    2577 generic.go:356] "Generic (PLEG): container finished" podID="cb8251a8-f48e-4af4-9606-5cbdea4a6b96" containerID="3e4987959f519a673b4f37360768705d0d07aecd81e00d5927ff05fa9d616de9" exitCode=0
Nov 03 19:26:23.821133 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:23.821020    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5hhz" event={"ID":"cb8251a8-f48e-4af4-9606-5cbdea4a6b96","Type":"ContainerDied","Data":"3e4987959f519a673b4f37360768705d0d07aecd81e00d5927ff05fa9d616de9"}
Nov 03 19:26:23.860231 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:23.860125    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-pc8t6" podStartSLOduration=2.744508002 podStartE2EDuration="10.860091155s" totalImagesPullingTime="8.115583153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:23.859601274 +0000 UTC m=+11.708815378" watchObservedRunningTime="2026-11-03 19:26:23.860091155 +0000 UTC m=+11.709305233"
Nov 03 19:26:23.860784 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:23.860749    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-r2j57" podStartSLOduration=2.346759436 podStartE2EDuration="10.860739837s" totalImagesPullingTime="8.513980401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:23.836208236 +0000 UTC m=+11.685422326" watchObservedRunningTime="2026-11-03 19:26:23.860739837 +0000 UTC m=+11.709953917"
Nov 03 19:26:24.615728 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:24.615690    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:24.616013 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:24.615737    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:24.616013 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:24.615858    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:24.616013 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:24.615949    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:24.826464 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:24.826384    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5" event={"ID":"f0efd5bb-5c99-4c84-a77f-d816995e6f79","Type":"ContainerStarted","Data":"789232cd78953ad4c771bc7b653675fd4095cda0112f5dee1ac03d764cab7cc4"}
Nov 03 19:26:24.828176 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:24.828144    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sp6pz" event={"ID":"7d79046e-eee4-4336-a852-131f013288a7","Type":"ContainerStarted","Data":"b931ae8d580515da097560d46ab6a1aab7d9acb881e632d3d9296d6c660a1a0c"}
Nov 03 19:26:24.830605 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:24.830574    2577 generic.go:356] "Generic (PLEG): container finished" podID="cb8251a8-f48e-4af4-9606-5cbdea4a6b96" containerID="da56acb2d3195eef5e218bff82876a2408faf44e2073f177675a3ef7bba45e60" exitCode=0
Nov 03 19:26:24.830718 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:24.830629    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5hhz" event={"ID":"cb8251a8-f48e-4af4-9606-5cbdea4a6b96","Type":"ContainerDied","Data":"da56acb2d3195eef5e218bff82876a2408faf44e2073f177675a3ef7bba45e60"}
Nov 03 19:26:24.850281 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:24.850222    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5" podStartSLOduration=3.22919815 podStartE2EDuration="11.85020187s" totalImagesPullingTime="8.62100372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:24.849183702 +0000 UTC m=+12.698397782" watchObservedRunningTime="2026-11-03 19:26:24.85020187 +0000 UTC m=+12.699415951"
Nov 03 19:26:24.894879 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:24.894800    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sp6pz" podStartSLOduration=3.6570497360000003 podStartE2EDuration="11.894787549s" totalImagesPullingTime="8.237737813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:24.894462907 +0000 UTC m=+12.743676987" watchObservedRunningTime="2026-11-03 19:26:24.894787549 +0000 UTC m=+12.744001629"
Nov 03 19:26:25.615146 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:25.615106    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:25.615339 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:25.615312    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:25.837014 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:25.836978    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6ksh4" event={"ID":"8273a1a4-5dbb-4f54-8f4c-4207cc11c228","Type":"ContainerStarted","Data":"6e726adb1a18b7a2fc879c57b4c80d20bd87633e56506360caf2d0d86aae54b0"}
Nov 03 19:26:25.839734 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:25.839703    2577 generic.go:356] "Generic (PLEG): container finished" podID="cb8251a8-f48e-4af4-9606-5cbdea4a6b96" containerID="ccc3e5b69a4cef78aaf623226e596d58981c7ef821095a878ee8b5f5ea46a4a4" exitCode=0
Nov 03 19:26:25.839861 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:25.839737    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5hhz" event={"ID":"cb8251a8-f48e-4af4-9606-5cbdea4a6b96","Type":"ContainerDied","Data":"ccc3e5b69a4cef78aaf623226e596d58981c7ef821095a878ee8b5f5ea46a4a4"}
Nov 03 19:26:25.858352 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:25.858290    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6ksh4" podStartSLOduration=2.331362637 podStartE2EDuration="12.858276493s" totalImagesPullingTime="10.526913856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:25.85788768 +0000 UTC m=+13.707101759" watchObservedRunningTime="2026-11-03 19:26:25.858276493 +0000 UTC m=+13.707490572"
Nov 03 19:26:26.615289 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:26.615252    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:26.615483 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:26.615309    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:26.615483 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:26.615428    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:26.615591 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:26.615561    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:27.615260 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:27.615219    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:27.616054 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:27.615369    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:28.615170 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:28.614927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:28.615351 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:28.614991    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:28.615351 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:28.615278    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:28.615718 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:28.615387    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:28.850462 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:28.850429    2577 generic.go:356] "Generic (PLEG): container finished" podID="cb8251a8-f48e-4af4-9606-5cbdea4a6b96" containerID="069d675080f58a4373ae69c56bc2ee0568255d72df3c2ddcf068ae46ba8aec9d" exitCode=0
Nov 03 19:26:28.850633 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:28.850469    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5hhz" event={"ID":"cb8251a8-f48e-4af4-9606-5cbdea4a6b96","Type":"ContainerDied","Data":"069d675080f58a4373ae69c56bc2ee0568255d72df3c2ddcf068ae46ba8aec9d"}
Nov 03 19:26:29.422928 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:29.422895    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:29.606177 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:29.606142    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:29.615188 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:29.615165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:29.615276 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:29.615262    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:29.855129 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:29.855097    2577 generic.go:356] "Generic (PLEG): container finished" podID="cb8251a8-f48e-4af4-9606-5cbdea4a6b96" containerID="5c94f271c8cfb62fa97befdf82c7990eeb4232287d75c5b5c89da0eab04458ae" exitCode=0
Nov 03 19:26:29.855502 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:29.855151    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5hhz" event={"ID":"cb8251a8-f48e-4af4-9606-5cbdea4a6b96","Type":"ContainerDied","Data":"5c94f271c8cfb62fa97befdf82c7990eeb4232287d75c5b5c89da0eab04458ae"}
Nov 03 19:26:30.290168 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:30.289972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs\") pod \"network-metrics-daemon-tz6b2\" (UID: \"7750339b-7420-4dda-904e-8221ba6c7ffe\") " pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:30.290272 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:30.290087    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:26:30.290316 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:30.290285    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs podName:7750339b-7420-4dda-904e-8221ba6c7ffe nodeName:}" failed. No retries permitted until 2026-11-03 19:26:46.290267293 +0000 UTC m=+34.139481363 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs") pod "network-metrics-daemon-tz6b2" (UID: "7750339b-7420-4dda-904e-8221ba6c7ffe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:26:30.391337 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:30.391243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:30.391468 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:30.391359    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:26:30.391468 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:30.391422    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret podName:7b22e16a-511b-4d70-a39c-16160ef790b1 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:46.391409697 +0000 UTC m=+34.240623755 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret") pod "global-pull-secret-syncer-lqnwg" (UID: "7b22e16a-511b-4d70-a39c-16160ef790b1") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:26:30.491935 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:30.491875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl729\" (UniqueName: \"kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729\") pod \"network-check-target-494mb\" (UID: \"aef23c0d-d927-4a6e-bc22-288027b498a8\") " pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:30.492087 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:30.492050    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:26:30.492087 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:30.492072    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:26:30.492087 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:30.492082    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hl729 for pod openshift-network-diagnostics/network-check-target-494mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:26:30.492197 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:30.492133    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729 podName:aef23c0d-d927-4a6e-bc22-288027b498a8 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:46.49211901 +0000 UTC m=+34.341333067 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hl729" (UniqueName: "kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729") pod "network-check-target-494mb" (UID: "aef23c0d-d927-4a6e-bc22-288027b498a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:26:30.615892 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:30.615859    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:30.616051 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:30.615981    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:30.616051 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:30.616028    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:30.616185 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:30.616113    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:30.860577 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:30.860545    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-52ft7"
Nov 03 19:26:30.861084 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:30.860671    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:26:30.861139 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:30.861094    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-52ft7"
Nov 03 19:26:30.861216 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:30.861195    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5hhz" event={"ID":"cb8251a8-f48e-4af4-9606-5cbdea4a6b96","Type":"ContainerStarted","Data":"b2f05bdc5e2262c9376880fdda92036cbaaa53ae67239add706e56a2b81aa9a7"}
Nov 03 19:26:30.864083 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:30.864058    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8" event={"ID":"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4","Type":"ContainerStarted","Data":"18792fa72809cddc3e96867e2296824e0177bf2a2ae64aa0e9761aa48737a8b2"}
Nov 03 19:26:30.864197 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:30.864112    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8" event={"ID":"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4","Type":"ContainerStarted","Data":"18eae558fda7a622c8f6b68fb869b686e25e216017c70f876467d1051b1682a4"}
Nov 03 19:26:30.864197 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:30.864141    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8" event={"ID":"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4","Type":"ContainerStarted","Data":"e3c07441979c145b316eba048a6f1e6aed07c13cb38752bba7a25ef4f19bce3e"}
Nov 03 19:26:30.864197 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:30.864170    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8" event={"ID":"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4","Type":"ContainerStarted","Data":"b7fe3471e463d828889588247885b82827769482fdfd1fb5b0239f60d084a8c6"}
Nov 03 19:26:30.864197 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:30.864196    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8" event={"ID":"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4","Type":"ContainerStarted","Data":"a69cc0752e4784483cfe041c731e6d4eb64f33719fac53a7dba333f718621561"}
Nov 03 19:26:30.864314 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:30.864221    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8" event={"ID":"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4","Type":"ContainerStarted","Data":"1271c501999966565f8fe2ef9d107b515aac5ef0abd748af14885a1937d0612e"}
Nov 03 19:26:30.910540 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:30.910491    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-s5hhz" podStartSLOduration=6.736098767 podStartE2EDuration="17.910479159s" totalImagesPullingTime="11.174380392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:30.909973701 +0000 UTC m=+18.759187779" watchObservedRunningTime="2026-11-03 19:26:30.910479159 +0000 UTC m=+18.759693238"
Nov 03 19:26:30.989770 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:30.989730    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:31.126240 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:31.126168    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:31.615594 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:31.615566    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:31.615748 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:31.615666    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:31.667029 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:31.666996    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:31.798013 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:31.797986    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:31.941121 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:31.941048    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:32.615982 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:32.615892    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:32.615982 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:32.615915    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:32.616154 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:32.615996    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:32.616154 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:32.616130    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:32.870272 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:32.870193    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8" event={"ID":"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4","Type":"ContainerStarted","Data":"1c8bb51afab8bbbb83f6d80e712c167203263567cf4f0de0dece6c6dcd5b2e76"}
Nov 03 19:26:33.615619 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:33.615585    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:33.616081 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:33.615690    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:33.624074 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:33.624044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:33.788646 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:33.788610    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:33.927341 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:33.927265    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:34.135667 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:34.135635    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:34.309504 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:34.309474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:34.615585 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:34.615503    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:34.615585 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:34.615544    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:34.616390 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:34.615656    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:34.616390 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:34.615761    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:34.878056 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:34.877960    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8" event={"ID":"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4","Type":"ContainerStarted","Data":"fb11b57e7a598e695469387584843382e364d768219a8c6a9f2d618efddede53"}
Nov 03 19:26:34.878282 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:34.878157    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:34.878282 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:34.878228    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:34.878282 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:34.878272    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:34.894149 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:34.894121    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:34.894290 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:34.894222    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:26:34.922343 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:34.922293    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8" podStartSLOduration=6.773003109 podStartE2EDuration="21.922282249s" totalImagesPullingTime="15.14927914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:34.921966311 +0000 UTC m=+22.771180389" watchObservedRunningTime="2026-11-03 19:26:34.922282249 +0000 UTC m=+22.771496328"
Nov 03 19:26:34.934047 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:34.934021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:35.492564 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:35.492518    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:35.615520 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:35.615492    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:35.615683 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:35.615586    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:35.655739 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:35.655686    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:35.783876 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:35.783788    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:35.953182 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:35.953148    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:36.086038 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:36.086005    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:36.229609 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:36.229580    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:36.368278 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:36.368208    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:36.615069 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:36.615040    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:36.615224 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:36.615152    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:36.615224 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:36.615203    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:36.615323 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:36.615301    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:36.629880 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:36.629806    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r2j57_74bbf640-fdad-4295-8810-f471ee4bb4da/dns-node-resolver/0.log"
Nov 03 19:26:37.043913 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:37.043815    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:37.182498 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:37.182466    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:37.386748 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:37.386637    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:37.518757 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:37.518720    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:37.615711 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:37.615677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:37.615894 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:37.615830    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:37.629626 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:37.629606    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2n8jc_90836408-13df-4995-acec-b7bf40eeec84/node-ca/0.log"
Nov 03 19:26:37.657627 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:37.657562    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:37.830568 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:37.830536    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:38.208100 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:38.208068    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:38.340584 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:38.340553    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:38.615256 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:38.615194    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:38.615256 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:38.615250    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:38.615459 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:38.615333    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:38.615459 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:38.615442    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:38.723245 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:38.723217    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:39.034830 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:39.034801    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:39.439167 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:39.439091    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:39.615083 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:39.615047    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:39.615247 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:39.615152    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:39.633098 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:39.633069    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:39.824349 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:39.824320    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:40.240594 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:40.240516    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:40.615706 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:40.615674    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:40.616133 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:40.615782    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:40.616133 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:40.615831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:40.616133 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:40.615935    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:41.028826 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:41.028738    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:41.429889 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:41.429708    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:41.615095 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:41.615045    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:41.615293 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:41.615176    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:41.639293 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:41.639247    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:41.832367 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:41.832336    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:42.223406 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:42.223302    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:42.616849 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:42.616804    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:42.616997 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:42.616945    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:42.616997 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:42.616990    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:42.617110 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:42.617090    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:43.030309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:43.030208    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:43.425734 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:43.425705    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:43.615382 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:43.615354    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:43.615547 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:43.615446    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:43.631521 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:43.631492    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:43.835727 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:43.835699    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:44.094889 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:44.094783    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:44.235365 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:44.235327    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:44.363457 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:44.363361    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:44.615934 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:44.615848    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:44.615934 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:44.615887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:44.616112 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:44.615975    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:44.616112 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:44.616093    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:45.027492 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:45.027408    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:45.462789 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:45.462753    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:45.615746 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:45.615710    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:45.615957 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:45.615808    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:45.631260 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:45.631228    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:45.840395 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:45.840362    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:46.230431 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:46.230352    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:46.304617 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:46.304585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs\") pod \"network-metrics-daemon-tz6b2\" (UID: \"7750339b-7420-4dda-904e-8221ba6c7ffe\") " pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:46.304731 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:46.304706    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:26:46.304774 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:46.304760    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs podName:7750339b-7420-4dda-904e-8221ba6c7ffe nodeName:}" failed. No retries permitted until 2026-11-03 19:27:18.30474604 +0000 UTC m=+66.153960098 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs") pod "network-metrics-daemon-tz6b2" (UID: "7750339b-7420-4dda-904e-8221ba6c7ffe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:26:46.405915 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:46.405879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:46.406042 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:46.406023    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:26:46.406097 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:46.406088    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret podName:7b22e16a-511b-4d70-a39c-16160ef790b1 nodeName:}" failed. No retries permitted until 2026-11-03 19:27:18.40607229 +0000 UTC m=+66.255286348 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret") pod "global-pull-secret-syncer-lqnwg" (UID: "7b22e16a-511b-4d70-a39c-16160ef790b1") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:26:46.426242 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:46.426214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:46.507087 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:46.507009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl729\" (UniqueName: \"kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729\") pod \"network-check-target-494mb\" (UID: \"aef23c0d-d927-4a6e-bc22-288027b498a8\") " pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:46.507425 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:46.507165    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:26:46.507425 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:46.507186    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:26:46.507425 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:46.507196    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hl729 for pod openshift-network-diagnostics/network-check-target-494mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:26:46.507425 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:46.507245    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729 podName:aef23c0d-d927-4a6e-bc22-288027b498a8 nodeName:}" failed. No retries permitted until 2026-11-03 19:27:18.507233224 +0000 UTC m=+66.356447295 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hl729" (UniqueName: "kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729") pod "network-check-target-494mb" (UID: "aef23c0d-d927-4a6e-bc22-288027b498a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:26:46.615823 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:46.615780    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:46.615997 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:46.615913    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:46.615997 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:46.615963    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:46.616076 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:46.616061    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:46.893025 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:46.892981    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:47.022965 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:47.022936    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:47.615783 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:47.615748    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:47.616177 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:47.615861    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:48.615562 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:48.615524    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:48.615736 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:48.615663    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:48.615736 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:48.615701    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:48.615823 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:48.615794    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:49.615880 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:49.615848    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:49.616337 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:49.615968    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:50.615044 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:50.615005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:50.615241 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:50.615117    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:50.615241 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:50.615167    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:50.615312 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:50.615253    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:50.902319 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:50.902242    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:51.035990 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:51.035923    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:51.615675 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:51.615645    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:51.615867 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:51.615750    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:52.616001 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:52.615966    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:52.616465 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:52.616080    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:52.616465 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:52.616122    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:52.616465 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:52.616222    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:53.615490 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:53.615462    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:53.615658 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:53.615556    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:53.709766 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:53.709737    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:53.836920 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:53.836888    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:54.036873 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:54.036849    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:54.169437 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:54.169406    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:54.297351 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:54.297274    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:54.615353 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:54.615280    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:54.615353 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:54.615311    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:54.615553 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:54.615411    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:54.615553 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:54.615437    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:55.615446 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:55.615416    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:55.615871 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:55.615514    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:56.540386 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:56.540349    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:56.614912 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:56.614884    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:56.615077 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:56.614999    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:56.615077 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:56.615047    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:56.615192 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:56.615127    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:56.730324 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:56.730287    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:56.883547 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:56.883473    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:57.341280 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:57.341247    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tz6b2"]
Nov 03 19:26:57.341427 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:57.341400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:57.341548 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:57.341524    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:57.344636 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:57.344609    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-494mb"]
Nov 03 19:26:57.344961 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:57.344929    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:57.345082 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:57.345045    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:57.345321 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:57.345301    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lqnwg"]
Nov 03 19:26:57.345413 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:57.345399    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:57.345512 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:57.345495    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:58.615551 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:58.615518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:26:58.615551 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:58.615547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:26:58.616100 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:58.615518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:26:58.616100 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:58.615649    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:26:58.616100 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:58.615708    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:26:58.616100 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:26:58.615807    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:26:59.394188 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:59.394160    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:59.562273 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:26:59.562232    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:00.049423 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:00.049387    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:00.176365 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:00.176315    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:00.615133 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:00.615102    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:27:00.615316 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:00.615217    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:27:00.615316 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:27:00.615267    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-494mb" podUID="aef23c0d-d927-4a6e-bc22-288027b498a8"
Nov 03 19:27:00.615316 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:00.615303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:27:00.615451 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:27:00.615280    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqnwg" podUID="7b22e16a-511b-4d70-a39c-16160ef790b1"
Nov 03 19:27:00.615451 ip-10-0-133-29 kubenswrapper[2577]: E1103 19:27:00.615374    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz6b2" podUID="7750339b-7420-4dda-904e-8221ba6c7ffe"
Nov 03 19:27:01.231059 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:01.231024    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:01.366724 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:01.366694    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:01.499656 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:01.499547    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:01.843004 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:01.842975    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:01.900277 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:01.900243    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeReady"
Nov 03 19:27:01.900466 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:01.900400    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 19:27:01.983197 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:01.983163    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hkjp4"]
Nov 03 19:27:01.985264 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:01.985243    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:01.992666 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:01.992617    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 19:27:01.992666 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:01.992620    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 19:27:01.992666 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:01.992622    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 19:27:01.992666 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:01.992642    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 19:27:01.992909 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:01.992720    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gtgvx\""
Nov 03 19:27:02.003107 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.003085    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hkjp4"]
Nov 03 19:27:02.007787 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.007763    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h6nf4"]
Nov 03 19:27:02.009844 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.009821    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:02.026572 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.026541    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h6nf4"]
Nov 03 19:27:02.036704 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.036670    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:02.071613 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.071579    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xszjb"]
Nov 03 19:27:02.073260 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.073245    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xszjb"
Nov 03 19:27:02.088428 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.088399    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xszjb"]
Nov 03 19:27:02.092874 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.092852    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 19:27:02.093084 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.093036    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 19:27:02.093174 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.093162    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 19:27:02.093354 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.093340    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 19:27:02.093455 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.093442    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 19:27:02.095282 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.095266    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6b2s9\""
Nov 03 19:27:02.095365 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.095330    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-qhgjf\""
Nov 03 19:27:02.112600 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.112569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f6a7918-e7d4-495f-9514-82e883318c90-data-volume\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:02.112600 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.112600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f6a7918-e7d4-495f-9514-82e883318c90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:02.112750 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.112626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea78abe-c045-41df-b4c8-d5275dcb19e4-config-volume\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:02.112750 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.112675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2f6a7918-e7d4-495f-9514-82e883318c90-crio-socket\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:02.112816 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.112760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea78abe-c045-41df-b4c8-d5275dcb19e4-tmp-dir\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:02.112816 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.112778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ea78abe-c045-41df-b4c8-d5275dcb19e4-metrics-tls\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:02.112816 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.112800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlkr2\" (UniqueName: \"kubernetes.io/projected/4ea78abe-c045-41df-b4c8-d5275dcb19e4-kube-api-access-xlkr2\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:02.112923 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.112862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f6a7918-e7d4-495f-9514-82e883318c90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:02.112955 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.112929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lthnm\" (UniqueName: \"kubernetes.io/projected/2f6a7918-e7d4-495f-9514-82e883318c90-kube-api-access-lthnm\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:02.213969 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.213932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24476d57-ccbb-45a7-a594-5ba46b48ea41-cert\") pod \"ingress-canary-xszjb\" (UID: \"24476d57-ccbb-45a7-a594-5ba46b48ea41\") " pod="openshift-ingress-canary/ingress-canary-xszjb"
Nov 03 19:27:02.213969 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.213972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f6a7918-e7d4-495f-9514-82e883318c90-data-volume\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:02.214140 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.213989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f6a7918-e7d4-495f-9514-82e883318c90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:02.214140 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.214008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea78abe-c045-41df-b4c8-d5275dcb19e4-config-volume\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:02.214140 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.214072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twxzs\" (UniqueName: \"kubernetes.io/projected/24476d57-ccbb-45a7-a594-5ba46b48ea41-kube-api-access-twxzs\") pod \"ingress-canary-xszjb\" (UID: \"24476d57-ccbb-45a7-a594-5ba46b48ea41\") " pod="openshift-ingress-canary/ingress-canary-xszjb"
Nov 03 19:27:02.214140 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.214110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2f6a7918-e7d4-495f-9514-82e883318c90-crio-socket\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:02.214328 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.214157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea78abe-c045-41df-b4c8-d5275dcb19e4-tmp-dir\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:02.214328 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.214179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ea78abe-c045-41df-b4c8-d5275dcb19e4-metrics-tls\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:02.214328 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.214254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlkr2\" (UniqueName: \"kubernetes.io/projected/4ea78abe-c045-41df-b4c8-d5275dcb19e4-kube-api-access-xlkr2\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:02.214328 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.214294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f6a7918-e7d4-495f-9514-82e883318c90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:02.214328 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.214325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lthnm\" (UniqueName: \"kubernetes.io/projected/2f6a7918-e7d4-495f-9514-82e883318c90-kube-api-access-lthnm\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:02.214554 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.214426    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2f6a7918-e7d4-495f-9514-82e883318c90-crio-socket\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:02.214645 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.214622    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f6a7918-e7d4-495f-9514-82e883318c90-data-volume\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:02.214737 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.214719    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea78abe-c045-41df-b4c8-d5275dcb19e4-tmp-dir\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:02.214951 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.214932    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f6a7918-e7d4-495f-9514-82e883318c90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:02.215006 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.214932    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea78abe-c045-41df-b4c8-d5275dcb19e4-config-volume\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:02.217851 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.217816    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f6a7918-e7d4-495f-9514-82e883318c90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:02.217953 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.217817    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ea78abe-c045-41df-b4c8-d5275dcb19e4-metrics-tls\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:02.227928 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.227906    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lthnm\" (UniqueName: \"kubernetes.io/projected/2f6a7918-e7d4-495f-9514-82e883318c90-kube-api-access-lthnm\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:02.228032 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.227934    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlkr2\" (UniqueName: \"kubernetes.io/projected/4ea78abe-c045-41df-b4c8-d5275dcb19e4-kube-api-access-xlkr2\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:02.297814 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.297774    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:02.315411 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.315382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24476d57-ccbb-45a7-a594-5ba46b48ea41-cert\") pod \"ingress-canary-xszjb\" (UID: \"24476d57-ccbb-45a7-a594-5ba46b48ea41\") " pod="openshift-ingress-canary/ingress-canary-xszjb"
Nov 03 19:27:02.315522 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.315423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twxzs\" (UniqueName: \"kubernetes.io/projected/24476d57-ccbb-45a7-a594-5ba46b48ea41-kube-api-access-twxzs\") pod \"ingress-canary-xszjb\" (UID: \"24476d57-ccbb-45a7-a594-5ba46b48ea41\") " pod="openshift-ingress-canary/ingress-canary-xszjb"
Nov 03 19:27:02.317679 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.317657    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24476d57-ccbb-45a7-a594-5ba46b48ea41-cert\") pod \"ingress-canary-xszjb\" (UID: \"24476d57-ccbb-45a7-a594-5ba46b48ea41\") " pod="openshift-ingress-canary/ingress-canary-xszjb"
Nov 03 19:27:02.320528 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.320503    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:02.344602 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.344522    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxzs\" (UniqueName: \"kubernetes.io/projected/24476d57-ccbb-45a7-a594-5ba46b48ea41-kube-api-access-twxzs\") pod \"ingress-canary-xszjb\" (UID: \"24476d57-ccbb-45a7-a594-5ba46b48ea41\") " pod="openshift-ingress-canary/ingress-canary-xszjb"
Nov 03 19:27:02.382857 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.382351    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xszjb"
Nov 03 19:27:02.587638 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.587600    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hkjp4"]
Nov 03 19:27:02.603465 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.603439    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h6nf4"]
Nov 03 19:27:02.606114 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:27:02.606082    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea78abe_c045_41df_b4c8_d5275dcb19e4.slice/crio-dfad3869bcc60f8c770c83b314970aa32b86a84c9288a269f6182ff3047dadf3 WatchSource:0}: Error finding container dfad3869bcc60f8c770c83b314970aa32b86a84c9288a269f6182ff3047dadf3: Status 404 returned error can't find the container with id dfad3869bcc60f8c770c83b314970aa32b86a84c9288a269f6182ff3047dadf3
Nov 03 19:27:02.620680 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.620653    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:27:02.620802 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.620656    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:27:02.620880 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.620807    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:27:02.658242 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.658212    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xszjb"]
Nov 03 19:27:02.661174 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:27:02.661145    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24476d57_ccbb_45a7_a594_5ba46b48ea41.slice/crio-4a354f240e69f14dcd397a9942991b43019e981cb3ec54b6911a2c609e1f75ec WatchSource:0}: Error finding container 4a354f240e69f14dcd397a9942991b43019e981cb3ec54b6911a2c609e1f75ec: Status 404 returned error can't find the container with id 4a354f240e69f14dcd397a9942991b43019e981cb3ec54b6911a2c609e1f75ec
Nov 03 19:27:02.702621 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.702585    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xlp4b\""
Nov 03 19:27:02.702621 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.702618    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5vlcb\""
Nov 03 19:27:02.702621 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.702614    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xv5d7\""
Nov 03 19:27:02.702937 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.702647    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:27:02.702937 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.702618    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:27:02.709412 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.709387    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:27:02.709554 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.709504    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:27:02.780626 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.780582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:02.938753 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.938673    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xszjb" event={"ID":"24476d57-ccbb-45a7-a594-5ba46b48ea41","Type":"ContainerStarted","Data":"4a354f240e69f14dcd397a9942991b43019e981cb3ec54b6911a2c609e1f75ec"}
Nov 03 19:27:02.939853 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.939817    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h6nf4" event={"ID":"4ea78abe-c045-41df-b4c8-d5275dcb19e4","Type":"ContainerStarted","Data":"dfad3869bcc60f8c770c83b314970aa32b86a84c9288a269f6182ff3047dadf3"}
Nov 03 19:27:02.941229 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.941208    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hkjp4" event={"ID":"2f6a7918-e7d4-495f-9514-82e883318c90","Type":"ContainerStarted","Data":"a307896b4e6dd62f89f9ac5ed80e8e6d2fd9ac2d01adf217b7e5fa537059a003"}
Nov 03 19:27:02.941334 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:02.941247    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hkjp4" event={"ID":"2f6a7918-e7d4-495f-9514-82e883318c90","Type":"ContainerStarted","Data":"d5ff45e7d5458dadc758ecfead2f4d8d6dfb94f8417418405e9e6928e059e64d"}
Nov 03 19:27:03.050872 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:03.047701    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:03.176265 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:03.176225    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:03.947034 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:03.946993    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hkjp4" event={"ID":"2f6a7918-e7d4-495f-9514-82e883318c90","Type":"ContainerStarted","Data":"01f7e069a8bda636552c2d38821cfd908cf2bf972a06c69fe6e95e7b3cb95ce1"}
Nov 03 19:27:04.625317 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:04.625289    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:04.816240 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:04.816206    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:04.851915 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:04.851877    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z24qr"]
Nov 03 19:27:04.856222 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:04.856195    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:04.921858 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:04.921810    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 19:27:04.922162 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:04.922142    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 19:27:04.922355 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:04.922335    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6lz2t\""
Nov 03 19:27:04.939365 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:04.939339    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 19:27:04.939506 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:04.939474    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 19:27:04.939552 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:04.939537    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 19:27:04.939607 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:04.939595    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 19:27:04.951648 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:04.951618    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h6nf4" event={"ID":"4ea78abe-c045-41df-b4c8-d5275dcb19e4","Type":"ContainerStarted","Data":"57185d03ae40fed191cf0bf4eef543160e76bb0c27c986e5cdd763f38f60110d"}
Nov 03 19:27:04.952104 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:04.951678    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h6nf4" event={"ID":"4ea78abe-c045-41df-b4c8-d5275dcb19e4","Type":"ContainerStarted","Data":"3f2759b699bd668a5cf5bf1b423fd76b531728a2d0daf22406a63df66b438447"}
Nov 03 19:27:04.952104 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:04.951731    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:04.974065 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:04.974021    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-h6nf4" podStartSLOduration=2.163733815 podStartE2EDuration="3.974008028s" totalImagesPullingTime="1.810274213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:04.973765271 +0000 UTC m=+52.822979351" watchObservedRunningTime="2026-11-03 19:27:04.974008028 +0000 UTC m=+52.823222118"
Nov 03 19:27:04.985025 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:04.984981    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:05.037483 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.037454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql6s2\" (UniqueName: \"kubernetes.io/projected/65df60f4-e551-4351-a403-26305bece450-kube-api-access-ql6s2\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.037604 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.037493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65df60f4-e551-4351-a403-26305bece450-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.037604 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.037514    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-sys\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.037604 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.037530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/65df60f4-e551-4351-a403-26305bece450-node-exporter-textfile\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.037749 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.037684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/65df60f4-e551-4351-a403-26305bece450-node-exporter-accelerators-collector-config\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.037787 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.037748    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-root\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.037869 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.037851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/65df60f4-e551-4351-a403-26305bece450-node-exporter-tls\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.037947 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.037894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-node-exporter-wtmp\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.038026 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.037965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65df60f4-e551-4351-a403-26305bece450-metrics-client-ca\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.139269 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.139191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/65df60f4-e551-4351-a403-26305bece450-node-exporter-accelerators-collector-config\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.139269 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.139237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-root\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.139470 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.139329    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/65df60f4-e551-4351-a403-26305bece450-node-exporter-tls\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.139470 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.139355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-node-exporter-wtmp\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.139470 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.139378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65df60f4-e551-4351-a403-26305bece450-metrics-client-ca\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.139470 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.139392    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-root\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.139753 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.139400    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6s2\" (UniqueName: \"kubernetes.io/projected/65df60f4-e551-4351-a403-26305bece450-kube-api-access-ql6s2\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.139807 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.139791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65df60f4-e551-4351-a403-26305bece450-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.139866 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.139812    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/65df60f4-e551-4351-a403-26305bece450-node-exporter-accelerators-collector-config\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.139910 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.139862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-sys\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.139910 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.139888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/65df60f4-e551-4351-a403-26305bece450-node-exporter-textfile\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.140632 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.140123    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-node-exporter-wtmp\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.140632 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.140185    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-sys\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.140632 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.140270    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/65df60f4-e551-4351-a403-26305bece450-node-exporter-textfile\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.140632 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.140307    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65df60f4-e551-4351-a403-26305bece450-metrics-client-ca\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.150195 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.150169    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65df60f4-e551-4351-a403-26305bece450-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.150299 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.150237    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/65df60f4-e551-4351-a403-26305bece450-node-exporter-tls\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.152745 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.152724    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql6s2\" (UniqueName: \"kubernetes.io/projected/65df60f4-e551-4351-a403-26305bece450-kube-api-access-ql6s2\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.161305 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.161273    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:05.175539 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.175506    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:05.187461 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:27:05.187424    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65df60f4_e551_4351_a403_26305bece450.slice/crio-cde373413c9a6018e072b80978ab096f5f128fa20923519e39af64db2f8a3ef6 WatchSource:0}: Error finding container cde373413c9a6018e072b80978ab096f5f128fa20923519e39af64db2f8a3ef6: Status 404 returned error can't find the container with id cde373413c9a6018e072b80978ab096f5f128fa20923519e39af64db2f8a3ef6
Nov 03 19:27:05.293633 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.293583    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:05.551814 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.551788    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:05.761026 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.760936    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:05.903244 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.903195    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:05.934547 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.934514    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:27:05.938260 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.938237    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:05.947132 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.947115    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 19:27:05.948186 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.948162    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 19:27:05.948352 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.948321    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kw62n\""
Nov 03 19:27:05.948403 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.948352    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 19:27:05.950970 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.950950    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 19:27:05.951079 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.951002    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 19:27:05.951141 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.951128    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 19:27:05.951335 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.951305    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 19:27:05.951335 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.951305    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 19:27:05.951758 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.951731    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 19:27:05.955130 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.955101    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z24qr" event={"ID":"65df60f4-e551-4351-a403-26305bece450","Type":"ContainerStarted","Data":"cde373413c9a6018e072b80978ab096f5f128fa20923519e39af64db2f8a3ef6"}
Nov 03 19:27:05.958982 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.958930    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xszjb" event={"ID":"24476d57-ccbb-45a7-a594-5ba46b48ea41","Type":"ContainerStarted","Data":"d1cda6f404b5b782452feba185ee50612d09937d8728818c9a1e98142201290d"}
Nov 03 19:27:05.963993 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.963964    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hkjp4" event={"ID":"2f6a7918-e7d4-495f-9514-82e883318c90","Type":"ContainerStarted","Data":"0167d73a91cfddea18c2bb24ea26601cba275d928d4865cacb232f767faa9e65"}
Nov 03 19:27:05.998929 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:05.998887    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:27:06.046064 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.046031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-config-volume\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.046064 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.046070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.046280 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.046095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-config-out\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.046280 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.046165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.046280 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.046228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.046413 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.046355    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.046413 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.046383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.046475 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.046418    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.046506 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.046481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-web-config\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.046550 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.046523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.046598 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.046551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.046598 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.046588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.046663 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.046611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qjx5\" (UniqueName: \"kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-kube-api-access-8qjx5\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.058395 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.058347    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xszjb" podStartSLOduration=1.222153303 podStartE2EDuration="4.058330113s" totalImagesPullingTime="2.83617681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:06.057385101 +0000 UTC m=+53.906599181" watchObservedRunningTime="2026-11-03 19:27:06.058330113 +0000 UTC m=+53.907544205"
Nov 03 19:27:06.074142 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.074114    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:06.087985 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.087938    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hkjp4" podStartSLOduration=2.873834859 podStartE2EDuration="5.087922333s" totalImagesPullingTime="2.214087474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:27:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:06.087008622 +0000 UTC m=+53.936222702" watchObservedRunningTime="2026-11-03 19:27:06.087922333 +0000 UTC m=+53.937136410"
Nov 03 19:27:06.147429 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.147403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.147527 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.147436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.147527 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.147477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.147527 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.147505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qjx5\" (UniqueName: \"kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-kube-api-access-8qjx5\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.147978 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.147949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-config-volume\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.148098 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.147991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.148098 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.148020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-config-out\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.148098 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.148045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.148098 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.148070    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.148295 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.148160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.148295 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.148192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.148295 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.148229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.148295 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.148228    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.148488 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.148296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-web-config\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.149278 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.149231    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.149608 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.149582    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.150652 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.150623    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.151005 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.150961    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.151093 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.151033    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.151541 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.151447    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.152692 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.151698    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-config-out\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.152692 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.151721    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-config-volume\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.152692 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.151860    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.152692 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.151986    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-web-config\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.155737 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.155718    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.162323 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.162296    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qjx5\" (UniqueName: \"kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-kube-api-access-8qjx5\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.252135 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.252078    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:06.264371 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.264341    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:06.382794 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.382758    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:06.551488 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.551457    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:27:06.555924 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:27:06.555899    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0827774e_9562_487f_9c6e_0934ef8707ea.slice/crio-90383a53ed9782bf968aff14e6c06a34b754360b77ab2524c4a17df8fc66c857 WatchSource:0}: Error finding container 90383a53ed9782bf968aff14e6c06a34b754360b77ab2524c4a17df8fc66c857: Status 404 returned error can't find the container with id 90383a53ed9782bf968aff14e6c06a34b754360b77ab2524c4a17df8fc66c857
Nov 03 19:27:06.892783 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.892760    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:06.968090 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.968057    2577 generic.go:356] "Generic (PLEG): container finished" podID="65df60f4-e551-4351-a403-26305bece450" containerID="bf3956da4a274d1de32a562d0ee431df2c08a7da8c2d3509b1d9b4ed3a1899c7" exitCode=0
Nov 03 19:27:06.968564 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.968143    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z24qr" event={"ID":"65df60f4-e551-4351-a403-26305bece450","Type":"ContainerDied","Data":"bf3956da4a274d1de32a562d0ee431df2c08a7da8c2d3509b1d9b4ed3a1899c7"}
Nov 03 19:27:06.969364 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:06.969341    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0827774e-9562-487f-9c6e-0934ef8707ea","Type":"ContainerStarted","Data":"90383a53ed9782bf968aff14e6c06a34b754360b77ab2524c4a17df8fc66c857"}
Nov 03 19:27:07.175929 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:07.175828    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:07.304193 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:07.304153    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:07.877528 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:07.877489    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:07.975383 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:07.975344    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z24qr" event={"ID":"65df60f4-e551-4351-a403-26305bece450","Type":"ContainerStarted","Data":"e887696f6895ec3376a1e4e50927fde64c8e0602275e92f70b9d4e5ea4b48b2e"}
Nov 03 19:27:07.975859 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:07.975408    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z24qr" event={"ID":"65df60f4-e551-4351-a403-26305bece450","Type":"ContainerStarted","Data":"88833b497b06c424b1b15cfbd0d7e59fb4ddeda83d3c83284a7abec0c9f108c8"}
Nov 03 19:27:07.976964 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:07.976938    2577 generic.go:356] "Generic (PLEG): container finished" podID="0827774e-9562-487f-9c6e-0934ef8707ea" containerID="dcc926ed375a62f53970a52e5d65852dbdcb9cda977e99db05668588f549c204" exitCode=0
Nov 03 19:27:07.977075 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:07.976973    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0827774e-9562-487f-9c6e-0934ef8707ea","Type":"ContainerDied","Data":"dcc926ed375a62f53970a52e5d65852dbdcb9cda977e99db05668588f549c204"}
Nov 03 19:27:08.003307 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:08.003259    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z24qr" podStartSLOduration=2.997686417 podStartE2EDuration="4.003246707s" totalImagesPullingTime="937.693111ms" totalInitContainerRuntime="67.867179ms" isStatefulPod=true podCreationTimestamp="2026-11-03 19:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:08.001089674 +0000 UTC m=+55.850303753" watchObservedRunningTime="2026-11-03 19:27:08.003246707 +0000 UTC m=+55.852460780"
Nov 03 19:27:08.018410 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:08.018374    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:08.219325 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:08.219241    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:08.435934 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:08.435898    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:08.570501 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:08.570463    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:09.218487 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.218453    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7d8d7db6bc-592nt"]
Nov 03 19:27:09.221913 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.221889    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.226661 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.226638    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 19:27:09.226661 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.226646    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 19:27:09.232958 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.232935    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d8d7db6bc-592nt"]
Nov 03 19:27:09.250551 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.250517    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 19:27:09.250852 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.250815    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 19:27:09.250938 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.250879    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2720n911htvu6\""
Nov 03 19:27:09.251004 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.250946    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4sj62\""
Nov 03 19:27:09.276629 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.276599    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-secret-metrics-server-client-certs\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.276777 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.276635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b95482-e639-4d23-97a8-59941d9ecbe4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.276777 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.276729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f9b95482-e639-4d23-97a8-59941d9ecbe4-metrics-server-audit-profiles\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.276777 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.276761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-client-ca-bundle\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.276896 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.276787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f9b95482-e639-4d23-97a8-59941d9ecbe4-audit-log\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.276930 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.276903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdspl\" (UniqueName: \"kubernetes.io/projected/f9b95482-e639-4d23-97a8-59941d9ecbe4-kube-api-access-bdspl\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.276962 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.276940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-secret-metrics-server-tls\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.378094 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.378071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdspl\" (UniqueName: \"kubernetes.io/projected/f9b95482-e639-4d23-97a8-59941d9ecbe4-kube-api-access-bdspl\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.378194 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.378122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-secret-metrics-server-tls\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.378194 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.378177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-secret-metrics-server-client-certs\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.378288 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.378210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b95482-e639-4d23-97a8-59941d9ecbe4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.378288 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.378275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f9b95482-e639-4d23-97a8-59941d9ecbe4-metrics-server-audit-profiles\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.378387 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.378313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-client-ca-bundle\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.378387 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.378350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f9b95482-e639-4d23-97a8-59941d9ecbe4-audit-log\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.378776 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.378749    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f9b95482-e639-4d23-97a8-59941d9ecbe4-audit-log\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.379059 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.379034    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b95482-e639-4d23-97a8-59941d9ecbe4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.379350 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.379324    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f9b95482-e639-4d23-97a8-59941d9ecbe4-metrics-server-audit-profiles\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.381123 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.381099    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-secret-metrics-server-tls\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.381239 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.381198    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-secret-metrics-server-client-certs\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.381439 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.381417    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-client-ca-bundle\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.391400 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.391380    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdspl\" (UniqueName: \"kubernetes.io/projected/f9b95482-e639-4d23-97a8-59941d9ecbe4-kube-api-access-bdspl\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.534631 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.534603    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:09.682762 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.682715    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:09.816661 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.816626    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d8d7db6bc-592nt"]
Nov 03 19:27:09.819392 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:27:09.819362    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b95482_e639_4d23_97a8_59941d9ecbe4.slice/crio-b1bfbbe06579eaad0b3ea2580255b0879fa7ab032bc3a4994632eafc50d1686a WatchSource:0}: Error finding container b1bfbbe06579eaad0b3ea2580255b0879fa7ab032bc3a4994632eafc50d1686a: Status 404 returned error can't find the container with id b1bfbbe06579eaad0b3ea2580255b0879fa7ab032bc3a4994632eafc50d1686a
Nov 03 19:27:09.820560 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.820537    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:09.955564 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.955518    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:09.984405 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.984369    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt" event={"ID":"f9b95482-e639-4d23-97a8-59941d9ecbe4","Type":"ContainerStarted","Data":"b1bfbbe06579eaad0b3ea2580255b0879fa7ab032bc3a4994632eafc50d1686a"}
Nov 03 19:27:09.987327 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.987298    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0827774e-9562-487f-9c6e-0934ef8707ea","Type":"ContainerStarted","Data":"27b8a439469acba6bebe5dade52a297a012b9a930e9bf6d686fca8a2051184e2"}
Nov 03 19:27:09.987432 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.987351    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0827774e-9562-487f-9c6e-0934ef8707ea","Type":"ContainerStarted","Data":"eb2ae76c62df0d09b240aea79a3a7c77698e25c9406fbbfae9542b9810b9e285"}
Nov 03 19:27:09.987432 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.987377    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0827774e-9562-487f-9c6e-0934ef8707ea","Type":"ContainerStarted","Data":"7d6c8a7691de319d2ae6933044d9fdf35e430df69babdd9abb404944d4e1ae79"}
Nov 03 19:27:09.987432 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.987401    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0827774e-9562-487f-9c6e-0934ef8707ea","Type":"ContainerStarted","Data":"aa1f57f1d141703876722bc27cf46fc511beee32c35e05d749f518dd9a0cc484"}
Nov 03 19:27:09.987432 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:09.987424    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0827774e-9562-487f-9c6e-0934ef8707ea","Type":"ContainerStarted","Data":"91f015b7d832ff6c64a800d3cb643ea563f3c8be16d6463a28046a392d128f0d"}
Nov 03 19:27:10.996269 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:10.996219    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0827774e-9562-487f-9c6e-0934ef8707ea","Type":"ContainerStarted","Data":"e213c3ca058418f708f618a1c9b13d4ed8b2eee740394dc5e556ccfa08ec0977"}
Nov 03 19:27:11.043946 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:11.043888    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.830495495 podStartE2EDuration="6.043871745s" totalImagesPullingTime="3.199742436s" totalInitContainerRuntime="13.633814ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:11.040787751 +0000 UTC m=+58.890001901" watchObservedRunningTime="2026-11-03 19:27:11.043871745 +0000 UTC m=+58.893085824"
Nov 03 19:27:11.320012 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:11.319980    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:12.000707 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:12.000660    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt" event={"ID":"f9b95482-e639-4d23-97a8-59941d9ecbe4","Type":"ContainerStarted","Data":"c67ba964852e4ac01aff5bf82aae27c82a4d3bf04da604ade0132f1fbee650e0"}
Nov 03 19:27:12.029771 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:12.029722    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt" podStartSLOduration=1.691090556 podStartE2EDuration="3.029709668s" totalImagesPullingTime="1.338619112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:12.02900683 +0000 UTC m=+59.878220910" watchObservedRunningTime="2026-11-03 19:27:12.029709668 +0000 UTC m=+59.878923749"
Nov 03 19:27:12.303707 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:12.303673    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:12.437877 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:12.437802    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:14.966682 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:14.966652    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:16.320921 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:16.320885    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:16.451344 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:16.451317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:18.346209 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.346173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs\") pod \"network-metrics-daemon-tz6b2\" (UID: \"7750339b-7420-4dda-904e-8221ba6c7ffe\") " pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:27:18.349974 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.349953    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:27:18.359130 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.359099    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs\") pod \"network-metrics-daemon-tz6b2\" (UID: \"7750339b-7420-4dda-904e-8221ba6c7ffe\") " pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:27:18.446500 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.446444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:27:18.450285 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.450266    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:27:18.459464 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.459442    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:27:18.547224 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.547182    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl729\" (UniqueName: \"kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729\") pod \"network-check-target-494mb\" (UID: \"aef23c0d-d927-4a6e-bc22-288027b498a8\") " pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:27:18.550893 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.550862    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:27:18.556826 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.556803    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5vlcb\""
Nov 03 19:27:18.561219 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.561201    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:27:18.562258 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.562241    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xv5d7\""
Nov 03 19:27:18.563300 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.563285    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:27:18.569667 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.569646    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:27:18.570670 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.570651    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl729\" (UniqueName: \"kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729\") pod \"network-check-target-494mb\" (UID: \"aef23c0d-d927-4a6e-bc22-288027b498a8\") " pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:27:18.634968 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.634938    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:18.767011 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.766979    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:18.837896 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.837868    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lqnwg"]
Nov 03 19:27:18.840142 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:27:18.840114    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b22e16a_511b_4d70_a39c_16160ef790b1.slice/crio-b8c3873471347ae6a64e3c7bb47fb8c52b1eac1cc9aa1073b118f3ddf84dbf11 WatchSource:0}: Error finding container b8c3873471347ae6a64e3c7bb47fb8c52b1eac1cc9aa1073b118f3ddf84dbf11: Status 404 returned error can't find the container with id b8c3873471347ae6a64e3c7bb47fb8c52b1eac1cc9aa1073b118f3ddf84dbf11
Nov 03 19:27:18.844450 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.844419    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tz6b2"]
Nov 03 19:27:18.846793 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:27:18.846770    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7750339b_7420_4dda_904e_8221ba6c7ffe.slice/crio-8182981a1498c38ad6442dffbf172c877cdd1f029228ffdf3a3d04780e39e22c WatchSource:0}: Error finding container 8182981a1498c38ad6442dffbf172c877cdd1f029228ffdf3a3d04780e39e22c: Status 404 returned error can't find the container with id 8182981a1498c38ad6442dffbf172c877cdd1f029228ffdf3a3d04780e39e22c
Nov 03 19:27:18.852168 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.852148    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xlp4b\""
Nov 03 19:27:18.858328 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:18.858308    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:27:19.025150 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:19.025118    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tz6b2" event={"ID":"7750339b-7420-4dda-904e-8221ba6c7ffe","Type":"ContainerStarted","Data":"8182981a1498c38ad6442dffbf172c877cdd1f029228ffdf3a3d04780e39e22c"}
Nov 03 19:27:19.026235 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:19.026212    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lqnwg" event={"ID":"7b22e16a-511b-4d70-a39c-16160ef790b1","Type":"ContainerStarted","Data":"b8c3873471347ae6a64e3c7bb47fb8c52b1eac1cc9aa1073b118f3ddf84dbf11"}
Nov 03 19:27:19.129207 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:19.129137    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-494mb"]
Nov 03 19:27:19.132083 ip-10-0-133-29 kubenswrapper[2577]: W1103 19:27:19.132058    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef23c0d_d927_4a6e_bc22_288027b498a8.slice/crio-6fc781808bb612597caad79d5be6b03b2a2d23b053ec18f10994634aa005a7d9 WatchSource:0}: Error finding container 6fc781808bb612597caad79d5be6b03b2a2d23b053ec18f10994634aa005a7d9: Status 404 returned error can't find the container with id 6fc781808bb612597caad79d5be6b03b2a2d23b053ec18f10994634aa005a7d9
Nov 03 19:27:20.039364 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:20.039266    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-494mb" event={"ID":"aef23c0d-d927-4a6e-bc22-288027b498a8","Type":"ContainerStarted","Data":"6fc781808bb612597caad79d5be6b03b2a2d23b053ec18f10994634aa005a7d9"}
Nov 03 19:27:20.042934 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:20.042904    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tz6b2" event={"ID":"7750339b-7420-4dda-904e-8221ba6c7ffe","Type":"ContainerStarted","Data":"7532224fdd2b480b66c89cd7ea2f22ba56e68d92c101cfd349aac705b233870d"}
Nov 03 19:27:20.650015 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:20.649972    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:20.800352 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:20.800226    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:20.933746 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:20.933659    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:21.049293 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:21.049254    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tz6b2" event={"ID":"7750339b-7420-4dda-904e-8221ba6c7ffe","Type":"ContainerStarted","Data":"4014b1740916447cacf9b47aee7bb7a4597bb1c64ea8d29a99ad58a661c3dcf7"}
Nov 03 19:27:21.072309 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:21.072256    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tz6b2" podStartSLOduration=67.004489223 podStartE2EDuration="1m8.072238309s" totalImagesPullingTime="1.067749086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:21.06890001 +0000 UTC m=+68.918114090" watchObservedRunningTime="2026-11-03 19:27:21.072238309 +0000 UTC m=+68.921452390"
Nov 03 19:27:23.057987 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:23.057953    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-494mb" event={"ID":"aef23c0d-d927-4a6e-bc22-288027b498a8","Type":"ContainerStarted","Data":"84914e6aba4914865b75b2cb9b498da3f8821275ac8413d55c3448c597d9599a"}
Nov 03 19:27:23.058403 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:23.058111    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:27:23.083605 ip-10-0-133-29 kubenswrapper[2577]: I1103 19:27:23.083539    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-494mb" podStartSLOduration=66.97699296 podStartE2EDuration="1m10.083521761s" totalImagesPullingTime="3.106528801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:23.083216261 +0000 UTC m=+70.932430351" watchObservedRunningTime="2026-11-03 19:27:23.083521761 +0000 UTC m=+70.932735842"
Nov 03 19:27:23.685688 ip-10-0-133-29 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 19:27:23.704264 ip-10-0-133-29 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 19:27:23.704427 ip-10-0-133-29 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 19:27:23.704483 ip-10-0-133-29 systemd[1]: kubelet.service: Consumed 3.941s CPU time.
Nov 03 19:27:23.715383 ip-10-0-133-29 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 19:27:23.775885 ip-10-0-133-29 kubenswrapper[5237]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:27:23.775885 ip-10-0-133-29 kubenswrapper[5237]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 19:27:23.775885 ip-10-0-133-29 kubenswrapper[5237]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:27:23.775885 ip-10-0-133-29 kubenswrapper[5237]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:27:23.777982 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.777899    5237 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:27:23.777982 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.777981    5237 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:27:23.777982 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.777986    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.777990    5237 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.777994    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.777997    5237 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778000    5237 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778003    5237 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778006    5237 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778009    5237 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778012    5237 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778014    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778017    5237 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778020    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778023    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778025    5237 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778028    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778031    5237 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778033    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778036    5237 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778039    5237 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:27:23.778081 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778041    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778044    5237 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778047    5237 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778049    5237 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778052    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778054    5237 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778057    5237 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778068    5237 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778071    5237 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778073    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778076    5237 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778078    5237 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778081    5237 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778084    5237 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778087    5237 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778090    5237 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778093    5237 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778095    5237 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778098    5237 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778100    5237 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:27:23.778549 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778103    5237 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778106    5237 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778108    5237 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778111    5237 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778113    5237 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778116    5237 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778118    5237 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778121    5237 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778123    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778126    5237 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778129    5237 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778131    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778136    5237 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778140    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778144    5237 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778147    5237 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778151    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778153    5237 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778156    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778159    5237 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:27:23.779043 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778162    5237 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778164    5237 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778167    5237 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778169    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778172    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778175    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778177    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778180    5237 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778182    5237 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778185    5237 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778187    5237 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778190    5237 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778192    5237 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778195    5237 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778197    5237 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778200    5237 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778202    5237 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778205    5237 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778208    5237 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:27:23.779513 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778210    5237 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778214    5237 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778218    5237 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778221    5237 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778224    5237 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778227    5237 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778229    5237 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778232    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778236    5237 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778239    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778241    5237 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778244    5237 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778247    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778249    5237 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778253    5237 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778256    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778259    5237 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778262    5237 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778264    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:27:23.780029 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778267    5237 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778269    5237 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778272    5237 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778276    5237 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778278    5237 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778281    5237 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.778283    5237 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779064    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779071    5237 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779074    5237 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779076    5237 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779079    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779082    5237 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779085    5237 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779087    5237 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779090    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779093    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779095    5237 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779098    5237 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779101    5237 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:27:23.780484 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779104    5237 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779107    5237 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779109    5237 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779112    5237 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779115    5237 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779117    5237 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779120    5237 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779123    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779127    5237 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779130    5237 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779132    5237 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779135    5237 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779138    5237 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779140    5237 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779144    5237 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779148    5237 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779151    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779154    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779157    5237 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779159    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:27:23.780984 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779162    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779164    5237 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779167    5237 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779170    5237 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779172    5237 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779175    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779178    5237 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779181    5237 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779184    5237 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779187    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779190    5237 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779193    5237 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779195    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779198    5237 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779200    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779203    5237 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779206    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779208    5237 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779211    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:27:23.781474 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779214    5237 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779217    5237 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779220    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779223    5237 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779226    5237 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779228    5237 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779231    5237 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779233    5237 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779236    5237 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779238    5237 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779241    5237 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779244    5237 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779246    5237 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779249    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779251    5237 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779254    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779256    5237 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779259    5237 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779262    5237 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779265    5237 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:27:23.781945 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779267    5237 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779270    5237 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779273    5237 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779276    5237 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779278    5237 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779281    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779284    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779286    5237 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779290    5237 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779294    5237 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779297    5237 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779301    5237 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779303    5237 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779306    5237 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779324    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779328    5237 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779332    5237 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779336    5237 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:27:23.782417 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779339    5237 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779342    5237 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779344    5237 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779347    5237 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779350    5237 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779353    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779355    5237 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779358    5237 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779361    5237 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779363    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779366    5237 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779368    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779371    5237 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779374    5237 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779378    5237 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779380    5237 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779887    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779893    5237 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779896    5237 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779899    5237 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:27:23.782857 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779902    5237 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779904    5237 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779907    5237 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779910    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779912    5237 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779915    5237 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779918    5237 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779920    5237 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779923    5237 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779926    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779930    5237 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779932    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779935    5237 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779937    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779940    5237 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779943    5237 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779946    5237 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779948    5237 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779951    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:27:23.783343 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779953    5237 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779956    5237 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779959    5237 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779961    5237 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779964    5237 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779966    5237 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779969    5237 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779971    5237 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779974    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779977    5237 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779979    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779981    5237 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779984    5237 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779987    5237 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779989    5237 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779992    5237 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779995    5237 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.779997    5237 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780000    5237 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780003    5237 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:27:23.783861 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780005    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780008    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780011    5237 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780014    5237 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780017    5237 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780019    5237 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780022    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780024    5237 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780027    5237 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780029    5237 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780032    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780035    5237 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780037    5237 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780040    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780042    5237 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780045    5237 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780048    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780051    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780053    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:27:23.784353 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780056    5237 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780058    5237 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780061    5237 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780065    5237 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780068    5237 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780070    5237 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780073    5237 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780076    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780078    5237 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780081    5237 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780084    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780086    5237 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780088    5237 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780091    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780094    5237 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780096    5237 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780099    5237 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780102    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780104    5237 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:27:23.784805 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780108    5237 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780112    5237 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780115    5237 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780117    5237 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780120    5237 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780122    5237 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780125    5237 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780127    5237 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780129    5237 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780132    5237 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780135    5237 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780138    5237 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780140    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780143    5237 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780145    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780148    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780151    5237 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780153    5237 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780156    5237 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780158    5237 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:27:23.785289 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780161    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780163    5237 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780166    5237 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780170    5237 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780174    5237 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780267    5237 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780274    5237 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780283    5237 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780289    5237 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780296    5237 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780300    5237 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780308    5237 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780313    5237 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780320    5237 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780323    5237 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780326    5237 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780330    5237 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780333    5237 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780336    5237 flags.go:64] FLAG: --cgroup-root=""
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780339    5237 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780342    5237 flags.go:64] FLAG: --client-ca-file=""
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780345    5237 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780348    5237 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 19:27:23.785796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780353    5237 flags.go:64] FLAG: --cluster-domain=""
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780356    5237 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780359    5237 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780362    5237 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780365    5237 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780369    5237 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780372    5237 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780376    5237 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780379    5237 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780382    5237 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780385    5237 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780388    5237 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780391    5237 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780394    5237 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780398    5237 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780401    5237 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780405    5237 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780407    5237 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780411    5237 flags.go:64] FLAG: --enable-server="true"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780414    5237 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780420    5237 flags.go:64] FLAG: --event-burst="100"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780424    5237 flags.go:64] FLAG: --event-qps="50"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780431    5237 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780436    5237 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 19:27:23.786391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780439    5237 flags.go:64] FLAG: --eviction-hard=""
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780443    5237 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780447    5237 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780450    5237 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780453    5237 flags.go:64] FLAG: --eviction-soft=""
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780456    5237 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780459    5237 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780462    5237 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780465    5237 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780468    5237 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780471    5237 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780474    5237 flags.go:64] FLAG: --feature-gates=""
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780479    5237 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780482    5237 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780485    5237 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780489    5237 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780492    5237 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780495    5237 flags.go:64] FLAG: --help="false"
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780498    5237 flags.go:64] FLAG: --hostname-override="ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780502    5237 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780505    5237 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780508    5237 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780511    5237 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 19:27:23.787402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780514    5237 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780517    5237 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780520    5237 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780523    5237 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780526    5237 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780531    5237 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780534    5237 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780538    5237 flags.go:64] FLAG: --kube-reserved=""
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780541    5237 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780544    5237 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780548    5237 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780550    5237 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780553    5237 flags.go:64] FLAG: --lock-file=""
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780556    5237 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780559    5237 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780562    5237 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780568    5237 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780571    5237 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780574    5237 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780577    5237 flags.go:64] FLAG: --logging-format="text"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780580    5237 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780584    5237 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780586    5237 flags.go:64] FLAG: --manifest-url=""
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780589    5237 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780594    5237 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 19:27:23.788210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780597    5237 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780601    5237 flags.go:64] FLAG: --max-pods="110"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780604    5237 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780608    5237 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780611    5237 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780614    5237 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780617    5237 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780620    5237 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780623    5237 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780631    5237 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780634    5237 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780637    5237 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780641    5237 flags.go:64] FLAG: --pod-cidr=""
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780644    5237 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780648    5237 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780652    5237 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780655    5237 flags.go:64] FLAG: --port="10250"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780658    5237 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780661    5237 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0fef8e101635c0b98"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780664    5237 flags.go:64] FLAG: --qos-reserved=""
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780667    5237 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780671    5237 flags.go:64] FLAG: --register-node="true"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780674    5237 flags.go:64] FLAG: --register-with-taints=""
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780678    5237 flags.go:64] FLAG: --registry-burst="10"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780681    5237 flags.go:64] FLAG: --registry-qps="5"
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780683    5237 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 19:27:23.788815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780686    5237 flags.go:64] FLAG: --reserved-memory=""
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780690    5237 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780694    5237 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780697    5237 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780700    5237 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780703    5237 flags.go:64] FLAG: --runonce="false"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780706    5237 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780709    5237 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780712    5237 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780715    5237 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780718    5237 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780721    5237 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780724    5237 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780728    5237 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780731    5237 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780734    5237 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780736    5237 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780740    5237 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780745    5237 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780748    5237 flags.go:64] FLAG: --system-cgroups=""
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780752    5237 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780757    5237 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780761    5237 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780764    5237 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780767    5237 flags.go:64] FLAG: --tls-min-version=""
Nov 03 19:27:23.789479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780770    5237 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780774    5237 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780777    5237 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780780    5237 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780787    5237 flags.go:64] FLAG: --v="2"
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780792    5237 flags.go:64] FLAG: --version="false"
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780796    5237 flags.go:64] FLAG: --vmodule=""
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780803    5237 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.780806    5237 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780952    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780957    5237 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780960    5237 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780963    5237 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780966    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780968    5237 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780971    5237 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780973    5237 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780976    5237 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780979    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780981    5237 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780984    5237 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:27:23.790157 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780986    5237 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780991    5237 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.780996    5237 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781000    5237 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781004    5237 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781007    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781012    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781015    5237 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781018    5237 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781021    5237 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781024    5237 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781026    5237 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781029    5237 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781032    5237 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781035    5237 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781037    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781040    5237 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781042    5237 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781045    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:27:23.790670 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781048    5237 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781050    5237 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781053    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781055    5237 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781058    5237 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781061    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781063    5237 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781066    5237 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781069    5237 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781071    5237 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781074    5237 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781077    5237 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781079    5237 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781082    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781084    5237 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781087    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781090    5237 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781092    5237 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781095    5237 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:27:23.791189 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781099    5237 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781102    5237 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781104    5237 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781107    5237 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781110    5237 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781113    5237 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781116    5237 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781118    5237 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781121    5237 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781124    5237 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781126    5237 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781129    5237 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781131    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781134    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781136    5237 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781139    5237 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781141    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781144    5237 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781146    5237 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781149    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:27:23.791665 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781151    5237 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781154    5237 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781157    5237 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781159    5237 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781162    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781164    5237 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781182    5237 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781186    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781190    5237 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781192    5237 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781195    5237 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781198    5237 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781202    5237 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781205    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781208    5237 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781210    5237 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781213    5237 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781216    5237 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781219    5237 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781222    5237 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:27:23.792219 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781224    5237 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:27:23.792788 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781227    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:27:23.792788 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781229    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:27:23.792788 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781232    5237 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:27:23.792788 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781235    5237 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:27:23.792788 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781238    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:27:23.792788 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781240    5237 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:27:23.792788 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781243    5237 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:27:23.792788 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781246    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:27:23.792788 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781248    5237 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:27:23.792788 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781251    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:27:23.792788 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781253    5237 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:27:23.792788 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781256    5237 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:27:23.792788 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781259    5237 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:27:23.792788 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781261    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:27:23.792788 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781264    5237 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:27:23.792788 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.781289    5237 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781326    5237 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781331    5237 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781334    5237 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781337    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781340    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781342    5237 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781345    5237 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781347    5237 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781352    5237 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781354    5237 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781357    5237 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781360    5237 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781362    5237 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781368    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781371    5237 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781373    5237 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781376    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781379    5237 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:27:23.793208 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781381    5237 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781384    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781387    5237 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781389    5237 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781392    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781394    5237 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781397    5237 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781400    5237 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781402    5237 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781405    5237 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781407    5237 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781410    5237 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781413    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781415    5237 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781418    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781420    5237 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781423    5237 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781425    5237 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781428    5237 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:27:23.793663 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781431    5237 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781433    5237 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781436    5237 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781440    5237 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781442    5237 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781445    5237 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781448    5237 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781451    5237 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781459    5237 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781462    5237 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781465    5237 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781468    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781470    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781473    5237 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781476    5237 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781479    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781482    5237 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781485    5237 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781488    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781490    5237 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:27:23.794184 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781493    5237 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781496    5237 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781499    5237 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781501    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781504    5237 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781506    5237 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781509    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781512    5237 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781514    5237 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781517    5237 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781519    5237 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781522    5237 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781525    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781527    5237 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781530    5237 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781533    5237 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781536    5237 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781539    5237 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781542    5237 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:27:23.794734 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781544    5237 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781548    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781551    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781554    5237 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781557    5237 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781559    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781562    5237 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781565    5237 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781567    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781570    5237 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781573    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781575    5237 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781578    5237 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781580    5237 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781583    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781586    5237 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781589    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781591    5237 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781594    5237 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:27:23.795257 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781597    5237 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781599    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781602    5237 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781604    5237 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781607    5237 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781610    5237 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781612    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781614    5237 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781617    5237 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781621    5237 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.781626    5237 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.785673    5237 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.785688    5237 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.785706    5237 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.785712    5237 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785779    5237 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785790    5237 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785794    5237 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785797    5237 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:27:23.795711 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785800    5237 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785803    5237 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785814    5237 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785818    5237 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785822    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785824    5237 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785827    5237 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785830    5237 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785846    5237 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785850    5237 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785852    5237 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785855    5237 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785858    5237 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785861    5237 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785863    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785866    5237 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785869    5237 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785872    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785875    5237 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:27:23.796413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785877    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785880    5237 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785889    5237 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785893    5237 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785896    5237 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785899    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785901    5237 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785904    5237 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785907    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785909    5237 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785912    5237 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785921    5237 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785924    5237 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785940    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785944    5237 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785948    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785951    5237 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785953    5237 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785957    5237 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:27:23.796889 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785961    5237 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785965    5237 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785968    5237 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785971    5237 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785973    5237 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785976    5237 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785979    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785981    5237 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785984    5237 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785986    5237 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785990    5237 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785992    5237 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785995    5237 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.785998    5237 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786000    5237 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786003    5237 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786006    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786009    5237 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786011    5237 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786014    5237 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:27:23.797352 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786016    5237 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786019    5237 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786022    5237 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786024    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786027    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786036    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786039    5237 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786042    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786044    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786047    5237 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786050    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786052    5237 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786055    5237 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786058    5237 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786060    5237 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786063    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786066    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786069    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786071    5237 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786074    5237 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:27:23.797815 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786076    5237 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786079    5237 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786081    5237 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786084    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786086    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786089    5237 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786091    5237 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786094    5237 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786097    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786100    5237 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786103    5237 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786105    5237 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786108    5237 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786110    5237 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786113    5237 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786115    5237 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786118    5237 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786120    5237 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786128    5237 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786131    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:27:23.798306 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786134    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:27:23.798809 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786137    5237 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:27:23.798809 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786139    5237 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:27:23.798809 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786142    5237 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:27:23.798809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.786170    5237 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:27:23.798809 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.786937    5237 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:27:23.798809 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787034    5237 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:27:23.798809 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787039    5237 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:27:23.798809 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787046    5237 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:27:23.798809 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787053    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:27:23.798809 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787057    5237 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:27:23.798809 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787062    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:27:23.798809 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787072    5237 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:27:23.798809 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787076    5237 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:27:23.798809 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787080    5237 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:27:23.798809 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787085    5237 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:27:23.798809 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787089    5237 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787093    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787097    5237 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787103    5237 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787106    5237 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787111    5237 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787115    5237 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787119    5237 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787124    5237 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787132    5237 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787137    5237 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787141    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787146    5237 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787150    5237 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787154    5237 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787185    5237 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787190    5237 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787194    5237 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787199    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787203    5237 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:27:23.799207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787207    5237 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787211    5237 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787220    5237 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787224    5237 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787229    5237 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787233    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787238    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787242    5237 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787247    5237 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787251    5237 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787255    5237 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787259    5237 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787263    5237 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787267    5237 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787276    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787281    5237 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787286    5237 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787291    5237 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787295    5237 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:27:23.799672 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787300    5237 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787304    5237 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787309    5237 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787313    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787317    5237 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787322    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787326    5237 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787331    5237 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787340    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787371    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787410    5237 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787428    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787434    5237 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787440    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787445    5237 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787453    5237 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787460    5237 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787466    5237 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787471    5237 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:27:23.800137 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787476    5237 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787481    5237 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787486    5237 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787496    5237 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787500    5237 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787504    5237 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787509    5237 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787531    5237 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787536    5237 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787541    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787545    5237 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787549    5237 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787553    5237 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787559    5237 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787564    5237 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787578    5237 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787582    5237 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787585    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787727    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:27:23.800593 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787734    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:27:23.801151 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787737    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:27:23.801151 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787741    5237 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:27:23.801151 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787744    5237 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:27:23.801151 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787746    5237 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:27:23.801151 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787749    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:27:23.801151 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787752    5237 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:27:23.801151 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787755    5237 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:27:23.801151 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787758    5237 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:27:23.801151 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787761    5237 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:27:23.801151 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787764    5237 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:27:23.801151 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787766    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:27:23.801151 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787769    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:27:23.801151 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787772    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:27:23.801151 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787775    5237 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:27:23.801151 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787777    5237 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:27:23.801151 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787781    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:27:23.801151 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787784    5237 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:27:23.801548 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.787819    5237 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:27:23.801548 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787879    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:27:23.801548 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787885    5237 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:27:23.801548 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787888    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:27:23.801548 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787890    5237 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:27:23.801548 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787893    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:27:23.801548 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787896    5237 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:27:23.801548 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787900    5237 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:27:23.801548 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787904    5237 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:27:23.801548 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787912    5237 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:27:23.801548 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787915    5237 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:27:23.801548 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787918    5237 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:27:23.801548 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787920    5237 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:27:23.801548 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787923    5237 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:27:23.801548 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787925    5237 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787929    5237 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787932    5237 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787934    5237 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787937    5237 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787940    5237 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787942    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787945    5237 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787948    5237 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787950    5237 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787953    5237 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787957    5237 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787959    5237 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787962    5237 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787964    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787967    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787970    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787973    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787976    5237 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:27:23.801933 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787979    5237 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787981    5237 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787984    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787989    5237 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787992    5237 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787995    5237 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.787997    5237 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788002    5237 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788006    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788009    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788022    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788025    5237 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788027    5237 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788030    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788033    5237 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788036    5237 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788039    5237 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788041    5237 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788046    5237 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788050    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:27:23.802376 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788053    5237 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788056    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788059    5237 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788061    5237 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788064    5237 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788066    5237 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788069    5237 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788071    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788074    5237 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788077    5237 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788079    5237 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788082    5237 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788084    5237 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788087    5237 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788089    5237 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788093    5237 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788095    5237 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788098    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788100    5237 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788103    5237 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:27:23.802851 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788107    5237 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788110    5237 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788113    5237 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788122    5237 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788126    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788129    5237 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788131    5237 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788134    5237 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788137    5237 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788139    5237 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788142    5237 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788145    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788147    5237 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788150    5237 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788153    5237 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788155    5237 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788158    5237 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788161    5237 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788163    5237 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:27:23.803329 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788166    5237 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788168    5237 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788171    5237 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788174    5237 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788176    5237 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788179    5237 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788181    5237 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788184    5237 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788187    5237 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788189    5237 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788192    5237 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788194    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788199    5237 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788203    5237 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:23.788205    5237 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.788275    5237 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.789253    5237 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.789319    5237 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 19:27:23.803881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.789494    5237 server.go:1014] "Starting client certificate rotation"
Nov 03 19:27:23.804336 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.789638    5237 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:27:23.804336 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.789684    5237 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:21:12 +0000 UTC" deadline="2028-04-27 18:28:19.95619135 +0000 UTC"
Nov 03 19:27:23.804336 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.789719    5237 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12983h0m56.166474671s"
Nov 03 19:27:23.804336 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.789975    5237 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:27:23.804336 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.791106    5237 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:27:23.804336 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.792267    5237 log.go:25] "Validated CRI v1 runtime API"
Nov 03 19:27:23.804336 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.793980    5237 log.go:25] "Validated CRI v1 image API"
Nov 03 19:27:23.804336 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.794675    5237 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 19:27:23.804336 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.796545    5237 fs.go:135] Filesystem UUIDs: map[4068e963-221f-442e-a7da-88d1e13886b9:/dev/nvme0n1p3 66f1c584-2d12-4182-8a8b-590d47830b0f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 03 19:27:23.804580 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.796566    5237 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b7f51a479baf646f10794133fc19ecf46c03507fcd900f3f6ec74527c0069fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b7f51a479baf646f10794133fc19ecf46c03507fcd900f3f6ec74527c0069fb/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/396d324ef98b37e05f1034cfc35e813cc772e3cfd1065b4dbe33b1d2a2e71c9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/396d324ef98b37e05f1034cfc35e813cc772e3cfd1065b4dbe33b1d2a2e71c9a/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a354f240e69f14dcd397a9942991b43019e981cb3ec54b6911a2c609e1f75ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a354f240e69f14dcd397a9942991b43019e981cb3ec54b6911a2c609e1f75ec/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c6dedb710a0f80631462172893a46d7295f769277996791e99c4a9614d0998e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c6dedb710a0f80631462172893a46d7295f769277996791e99c4a9614d0998e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fc781808bb612597caad79d5be6b03b2a2d23b053ec18f10994634aa005a7d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fc781808bb612597caad79d5be6b03b2a2d23b053ec18f10994634aa005a7d9/userdata/shm major:0 minor:337 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8182981a1498c38ad6442dffbf172c877cdd1f029228ffdf3a3d04780e39e22c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8182981a1498c38ad6442dffbf172c877cdd1f029228ffdf3a3d04780e39e22c/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c116c6528d4b874734bde912eceb384ec550b0f7ecdc313fa3914e82e2d1db0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c116c6528d4b874734bde912eceb384ec550b0f7ecdc313fa3914e82e2d1db0/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90383a53ed9782bf968aff14e6c06a34b754360b77ab2524c4a17df8fc66c857/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90383a53ed9782bf968aff14e6c06a34b754360b77ab2524c4a17df8fc66c857/userdata/shm major:0 minor:268 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92ec458682e0252331cafbcd2ee616121af1ba6cd2a5e758a832c08908d25d10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92ec458682e0252331cafbcd2ee616121af1ba6cd2a5e758a832c08908d25d10/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98d32d8c2eb7304220d335c426e8ff22f09c940d9c5dd8aab874195c26b8df97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98d32d8c2eb7304220d335c426e8ff22f09c940d9c5dd8aab874195c26b8df97/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1bfbbe06579eaad0b3ea2580255b0879fa7ab032bc3a4994632eafc50d1686a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1bfbbe06579eaad0b3ea2580255b0879fa7ab032bc3a4994632eafc50d1686a/userdata/shm major:0 minor:307 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b66f962f7121d724fd502890519de74e9dbbf3005decd9e714b0a1c4bf68235d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b66f962f7121d724fd502890519de74e9dbbf3005decd9e714b0a1c4bf68235d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8c3873471347ae6a64e3c7bb47fb8c52b1eac1cc9aa1073b118f3ddf84dbf11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8c3873471347ae6a64e3c7bb47fb8c52b1eac1cc9aa1073b118f3ddf84dbf11/userdata/shm major:0 minor:329 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba441394447e92ee1a6f4075a70e010b40e3b0d1f12a3347950676421f6d7491/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba441394447e92ee1a6f4075a70e010b40e3b0d1f12a3347950676421f6d7491/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb65a1d8f94b24f8ced23ba2c44cc83e756d780b4b80025e2022ceba665fc1bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb65a1d8f94b24f8ced23ba2c44cc83e756d780b4b80025e2022ceba665fc1bd/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cde373413c9a6018e072b80978ab096f5f128fa20923519e39af64db2f8a3ef6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cde373413c9a6018e072b80978ab096f5f128fa20923519e39af64db2f8a3ef6/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5ff45e7d5458dadc758ecfead2f4d8d6dfb94f8417418405e9e6928e059e64d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5ff45e7d5458dadc758ecfead2f4d8d6dfb94f8417418405e9e6928e059e64d/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfad3869bcc60f8c770c83b314970aa32b86a84c9288a269f6182ff3047dadf3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfad3869bcc60f8c770c83b314970aa32b86a84c9288a269f6182ff3047dadf3/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed1e72c10959b9f9352dd0126173841cdfa6acce1eebd375ed73143110000e55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed1e72c10959b9f9352dd0126173841cdfa6acce1eebd375ed73143110000e55/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volume-subpaths/web-config/alertmanager/9 major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volume-subpaths/web-config/config-reloader/6 major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~empty-dir/config-out major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~projected/kube-api-access-8qjx5:{mountpoint:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~projected/kube-api-access-8qjx5 major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~projected/tls-assets major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/config-volume major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/web-config major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24476d57-ccbb-45a7-a594-5ba46b48ea41/volumes/kubernetes.io~projected/kube-api-access-twxzs:{mountpoint:/var/lib/kubelet/pods/24476d57-ccbb-45a7-a594-5ba46b48ea41/volumes/kubernetes.io~projected/kube-api-access-twxzs major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24476d57-ccbb-45a7-a594-5ba46b48ea41/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/24476d57-ccbb-45a7-a594-5ba46b48ea41/volumes/kubernetes.io~secret/cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f6a7918-e7d4-495f-9514-82e883318c90/volumes/kubernetes.io~projected/kube-api-access-lthnm:{mountpoint:/var/lib/kubelet/pods/2f6a7918-e7d4-495f-9514-82e883318c90/volumes/kubernetes.io~projected/kube-api-access-lthnm major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f6a7918-e7d4-495f-9514-82e883318c90/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/2f6a7918-e7d4-495f-9514-82e883318c90/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea78abe-c045-41df-b4c8-d5275dcb19e4/volumes/kubernetes.io~projected/kube-api-access-xlkr2:{mountpoint:/var/lib/kubelet/pods/4ea78abe-c045-41df-b4c8-d5275dcb19e4/volumes/kubernetes.io~projected/kube-api-access-xlkr2 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea78abe-c045-41df-b4c8-d5275dcb19e4/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/4ea78abe-c045-41df-b4c8-d5275dcb19e4/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65df60f4-e551-4351-a403-26305bece450/volumes/kubernetes.io~projected/kube-api-access-ql6s2:{mountpoint:/var/lib/kubelet/pods/65df60f4-e551-4351-a403-26305bece450/volumes/kubernetes.io~projected/kube-api-access-ql6s2 major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65df60f4-e551-4351-a403-26305bece450/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/65df60f4-e551-4351-a403-26305bece450/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65df60f4-e551-4351-a403-26305bece450/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/65df60f4-e551-4351-a403-26305bece450/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74bbf640-fdad-4295-8810-f471ee4bb4da/volumes/kubernetes.io~projected/kube-api-access-dmg4z:{mountpoint:/var/lib/kubelet/pods/74bbf640-fdad-4295-8810-f471ee4bb4da/volumes/kubernetes.io~projected/kube-api-access-dmg4z major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7750339b-7420-4dda-904e-8221ba6c7ffe/volumes/kubernetes.io~projected/kube-api-access-mjzq9:{mountpoint:/var/lib/kubelet/pods/7750339b-7420-4dda-904e-8221ba6c7ffe/volumes/kubernetes.io~projected/kube-api-access-mjzq9 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7750339b-7420-4dda-904e-8221ba6c7ffe/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/7750339b-7420-4dda-904e-8221ba6c7ffe/volumes/kubernetes.io~secret/metrics-certs major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b22e16a-511b-4d70-a39c-16160ef790b1/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7b22e16a-511b-4d70-a39c-16160ef790b1/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d79046e-eee4-4336-a852-131f013288a7/volumes/kubernetes.io~projected/kube-api-access-b45zq:{mountpoint:/var/lib/kubelet/pods/7d79046e-eee4-4336-a852-131f013288a7/volumes/kubernetes.io~projected/kube-api-access-b45zq major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8273a1a4-5dbb-4f54-8f4c-4207cc11c228/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8273a1a4-5dbb-4f54-8f4c-4207cc11c228/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8273a1a4-5dbb-4f54-8f4c-4207cc11c228/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8273a1a4-5dbb-4f54-8f4c-4207cc11c228/volumes/kubernetes.io~empty-dir/tmp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8273a1a4-5dbb-4f54-8f4c-4207cc11c228/volumes/kubernetes.io~projected/kube-api-access-grpgt:{mountpoint:/var/lib/kubelet/pods/8273a1a4-5dbb-4f54-8f4c-4207cc11c228/volumes/kubernetes.io~projected/kube-api-access-grpgt major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90836408-13df-4995-acec-b7bf40eeec84/volumes/kubernetes.io~projected/kube-api-access-v9fmg:{mountpoint:/var/lib/kubelet/pods/90836408-13df-4995-acec-b7bf40eeec84/volumes/kubernetes.io~projected/kube-api-access-v9fmg major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9095ae71-1326-4ec3-9755-164738a4da17/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/9095ae71-1326-4ec3-9755-164738a4da17/volumes/kubernetes.io~secret/agent-certs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98108a97-4bce-4291-8c25-ef6a88d542e4/volumes/kubernetes.io~projected/kube-api-access-np75z:{mountpoint:/var/lib/kubelet/pods/98108a97-4bce-4291-8c25-ef6a88d542e4/volumes/kubernetes.io~projected/kube-api-access-np75z major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4/volumes/kubernetes.io~projected/kube-api-access-r8hvm:{mountpoint:/var/lib/kubelet/pods/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4/volumes/kubernetes.io~projected/kube-api-access-r8hvm major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aef23c0d-d927-4a6e-bc22-288027b498a8/volumes/kubernetes.io~projected/kube-api-access-hl729:{mountpoint:/var/lib/kubelet/pods/aef23c0d-d927-4a6e-bc22-288027b498a8/volumes/kubernetes.io~projected/kube-api-access-hl729 major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb8251a8-f48e-4af4-9606-5cbdea4a6b96/volumes/kubernetes.io~projected/kube-api-access-fdkxd:{mountpoint:/var/lib/kubelet/pods/cb8251a8-f48e-4af4-9606-5cbdea4a6b96/volumes/kubernetes.io~projected/kube-api-access-fdkxd major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0efd5bb-5c99-4c84-a77f-d816995e6f79/volumes/kubernetes.io~projected/kube-api-access-cvttl:{mountpoint:/var/lib/kubelet/pods/f0efd5bb-5c99-4c84-a77f-d816995e6f79/volumes/kubernetes.io~projected/kube-api-access-cvttl major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b95482-e639-4d23-97a8-59941d9ecbe4/volumes/kubernetes.io~projected/kube-api-access-bdspl:{mountpoint:/var/lib/kubelet/pods/f9b95482-e639-4d23-97a8-59941d9ecbe4/volumes/kubernetes.io~projected/kube-api-access-bdspl major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b95482-e639-4d23-97a8-59941d9ecbe4/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/f9b95482-e639-4d23-97a8-59941d9ecbe4/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b95482-e639-4d23-97a8-59941d9ecbe4/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/f9b95482-e639-4d23-97a8-59941d9ecbe4/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b95482-e639-4d23-97a8-59941d9ecbe4/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/f9b95482-e639-4d23-97a8-59941d9ecbe4/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:288 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/727ca107328e1055ffd592267be44a9ef374fbd663361520a66a3ba25f5bf4f0/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d1afccbeb7ab5f1a8be17362d7c7ce43fa524f5fc5309291bd56aebc065513dc/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/cde7b5b77927b7b9e7bdcceac3384e4cc03e5f7460bd689a6df821885878ce6b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/3bc4c2c37bd7e1f894959be7ad176f2059c8af9a677da04e51d772bca9a98ca9/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/dcbde9aaa9d0b05d91722df1891d4987d87d31ad8009152104d83236dc908257/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5d0ae8ac5155b1236ea604a31e7c1c6dc9812646c3215feb19739976da883a9e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/29fc92de68380c20989df3be751ea860d3b451e513e7a08317333c37de5c2d1e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b8d41aa016461cc654d58a35a07ff630b3e8d393848ce2efdb8b281783355575/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/19c14521681195ca0e9f097b04582eabb77669babd98190ed9ace40bc275cc09/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/5e4acffac6cee3516a74ac730806a260e6a2bfa502c05c2be913a186595755c2/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/fa26c2955a3b9ceb4e2f69d7a91ea2816c6800a22597d16026410f8e2ea50ab4/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/18d8a93417cfc242b664e534ffce3f166b089d196400eeb3a1cc1b09b65f49b7/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/cd2b37ae263dce42031cf3eeaad0e8b71cc4d0f345202c82f457fb77958f5879/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/3487c7ce2aed0ac385fc8f94a6250c0b83014830365b190855ddca9fdb012c69/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e71cde2de22bde2288cbc7fbbcae801526537a6dceacfe47f023ef3aafbd9423/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/3621a0e2e30a044c8d172c2f4a5b7eef443636ce1f1fbc5332729525a5f0083e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4764ae3ec3f6d5ca45988a935032f07398d94039419d28a917f60f907a5a714a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/729962e87c403378b4ee01978f0a441facd0fc0af4d17273ae9d81dd4d5b0db0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/a1bfb492bfb9b0924817fe2ae319109381fcf7cea7617790d494b5cd2f905105/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/20201f88e274481fcc1fc557d6ef68960cf6d0ad826a2f9f316d10e1867c0df6/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/479d17f0d50e78478f32670fc75fea24e61f33a484de74f02a0e16c3212b1446/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/2d870d7bc4babceb5994507a29bbb492b3fb6715340d77f9e00a5236450a4624/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/0e13d96482cb633f2ab6ce840c7e561d66f6f866449ecdc78ee1b04dd7eeebbc/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/ea25487da4f5c44f0d3f888f69b1595f7f4785ffdfd173dbd8e1177e4f2d1a5c/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/11c538aa9c58665b2c5386a6ca6c36a82c3e64195e9f36fa03a8e0b9c328b0a6/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/38fef2a6886da746518ab893357c4bf668133403f8ee8c01756df1a96345bcf5/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/be2dbe3c884f4fe1adda47c141f2606ee5f13593bafd075e713ec0d53f525470/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/0cf64ad8bbd8de2dd1bc54ab9c12d23a021d0d6b3916d70e2b1098c4da2747ff/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/d23d92d615888f7b3a0504c0e4a57a7c7ecd73580b3bbe7f61e0360bcd01eee0/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/2a6eb5490b83f3d8ac3cbd457e39a127be4c3d2a4c42a69bc3f9d5218c8a4e19/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/1dd2114ee454dd689a228c3b6f02520822a0a49b0b0c024a2f15ca7d3ab91624/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/636dd1aa6945e2f78432bb4760f8371cd0b0dbb4dd3671e1f69cdf0e6c4da23c/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/2dec4411739b05c1c59a3372e1e6591b096e9ffccf66762c00dcd671426bf0ae/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/42864ff69e3e1eda7fc1028040c43001e9845116b044ea8c10349e1329d2262e/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/d839ae240e6304bf34838e6f39dfd84af8fb080ecb93c49c2af38a338844d6c6/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/bfab2060b5b38a2de9f6f77a68dfe53a97db7b41b048cf1268a742c32ead24db/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/eb3b3c9f525b282e376bdaccafeeae52853763c50583cc884db9181a52bc2b0c/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/80d367336ba5ff4824d0c8033e0c7683e12025c666c90f504625f738adacedee/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/2a2221f6163b2983ec44a8359c17d54d4ddae6bc544e416efb8acf6bc9e9cdc6/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/7c1c60da66bbe28b213a8a0936f6713541076ef405b57de2e7d66a91ba00e322/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/2318483122fe6791f9fb517ad2df72806bb9dde901bbe012019429b4e215fe19/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/4a2e5ebda516fe3ed932b8983f5ea3f57a021f20a50d520f7c19a494cc55eb14/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/535668b364769fcf124261128680fe81a76cd684619458ca93bf6f0f1e31e85e/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/1567e0746f5ecac4e09bda9a536b3802e1229893290ce623734b8898e521b59f/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/adeed6469589d983241e558b59b2bbf4c6e996ec205029d5613ba497b1683c58/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/fb5764a164dff8e743b3d4d46c748c88fcce3d019a0506b8015a75a815eb9f4a/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/225325b0b4a32609189e2042a61a3fd4b065255bb5e2c50e20bceea25567547e/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/e36de8e37490d58f260f739f062fbab347170da97ba686564c47b61bcfdd8697/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/b9d1e5ede357fe9ccdfc980d90df3519cb30206485a8c1884e3c4ddfb5bda7f1/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/64dac3308f401a9c91bd4939a45d7dde414d6ff8da5a93bc4bb9f4c57c54d261/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/d84fc7c90a326518c79583af4e22de7a7fd0132a5dd54ba423a624d0255e593f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/b6fc38abac6f216ee78ab8437df51c73ab0df0bbeaeb86da5b03ebc2d0437d03/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/57edaf0735d9f797b6d50f82ad478f13fea31135101d49275804f656eba88dd9/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/67799027eaa5953f20adba8f1c17389ea9c8654cab96082d229e09bc882ce5e8/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/39f1c524b8fc4c028271fadbf9c4a7529269e2074c5fdb26c5fb21d452b7c2a1/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/6b09b6dc0d777a3c4635852ef84ccd5c86bd4eb9ab4735439dddd2f15e7feb08/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/1bc4c4a6ef9f32767c0baf62f5a67c16adc7b941af92642f32bd274accf9ea61/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/daeed578a6ca43d55b627e77425426f7296e467b1e9bbb90cacddd060461370b/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/7398d795ccdfd88e6620be4e9c26f53e584eff4fcbb46b917087939fa71b7ce0/merged major:0 minor:98 fsType:overlay blockSize:0}]
Nov 03 19:27:23.804787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.804317    5237 manager.go:217] Machine: {Timestamp:2026-11-03 19:27:23.803846836 +0000 UTC m=+0.058154058 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099463 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f6604d39ed266e68c9437d0ed6480 SystemUUID:ec2f6604-d39e-d266-e68c-9437d0ed6480 BootID:de033993-ff93-41ed-b1c5-62b34ffc7442 Filesystems:[{Device:/run/containers/storage/overlay-containers/cde373413c9a6018e072b80978ab096f5f128fa20923519e39af64db2f8a3ef6/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea78abe-c045-41df-b4c8-d5275dcb19e4/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:266 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aef23c0d-d927-4a6e-bc22-288027b498a8/volumes/kubernetes.io~projected/kube-api-access-hl729 DeviceMajor:0 DeviceMinor:328 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/98108a97-4bce-4291-8c25-ef6a88d542e4/volumes/kubernetes.io~projected/kube-api-access-np75z DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f6a7918-e7d4-495f-9514-82e883318c90/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:265 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74bbf640-fdad-4295-8810-f471ee4bb4da/volumes/kubernetes.io~projected/kube-api-access-dmg4z DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98d32d8c2eb7304220d335c426e8ff22f09c940d9c5dd8aab874195c26b8df97/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/65df60f4-e551-4351-a403-26305bece450/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:241 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65df60f4-e551-4351-a403-26305bece450/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:245 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:263 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:267 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b95482-e639-4d23-97a8-59941d9ecbe4/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8273a1a4-5dbb-4f54-8f4c-4207cc11c228/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba441394447e92ee1a6f4075a70e010b40e3b0d1f12a3347950676421f6d7491/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:266 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/24476d57-ccbb-45a7-a594-5ba46b48ea41/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92ec458682e0252331cafbcd2ee616121af1ba6cd2a5e758a832c08908d25d10/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed1e72c10959b9f9352dd0126173841cdfa6acce1eebd375ed73143110000e55/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d79046e-eee4-4336-a852-131f013288a7/volumes/kubernetes.io~projected/kube-api-access-b45zq DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a354f240e69f14dcd397a9942991b43019e981cb3ec54b6911a2c609e1f75ec/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b22e16a-511b-4d70-a39c-16160ef790b1/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:327 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c116c6528d4b874734bde912eceb384ec550b0f7ecdc313fa3914e82e2d1db0/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b95482-e639-4d23-97a8-59941d9ecbe4/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f6a7918-e7d4-495f-9514-82e883318c90/volumes/kubernetes.io~projected/kube-api-access-lthnm DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4/volumes/kubernetes.io~projected/kube-api-access-r8hvm DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/9095ae71-1326-4ec3-9755-164738a4da17/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8182981a1498c38ad6442dffbf172c877cdd1f029228ffdf3a3d04780e39e22c/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c6dedb710a0f80631462172893a46d7295f769277996791e99c4a9614d0998e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfad3869bcc60f8c770c83b314970aa32b86a84c9288a269f6182ff3047dadf3/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/24476d57-ccbb-45a7-a594-5ba46b48ea41/volumes/kubernetes.io~projected/kube-api-access-twxzs DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:260 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/396d324ef98b37e05f1034cfc35e813cc772e3cfd1065b4dbe33b1d2a2e71c9a/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0efd5bb-5c99-4c84-a77f-d816995e6f79/volumes/kubernetes.io~projected/kube-api-access-cvttl DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea78abe-c045-41df-b4c8-d5275dcb19e4/volumes/kubernetes.io~projected/kube-api-access-xlkr2 DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7750339b-7420-4dda-904e-8221ba6c7ffe/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:322 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:259 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:266 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:267 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fc781808bb612597caad79d5be6b03b2a2d23b053ec18f10994634aa005a7d9/userdata/shm DeviceMajor:0 DeviceMinor:337 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90836408-13df-4995-acec-b7bf40eeec84/volumes/kubernetes.io~projected/kube-api-access-v9fmg DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8273a1a4-5dbb-4f54-8f4c-4207cc11c228/volumes/kubernetes.io~projected/kube-api-access-grpgt DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b7f51a479baf646f10794133fc19ecf46c03507fcd900f3f6ec74527c0069fb/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb65a1d8f94b24f8ced23ba2c44cc83e756d780b4b80025e2022ceba665fc1bd/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65df60f4-e551-4351-a403-26305bece450/volumes/kubernetes.io~projected/kube-api-access-ql6s2 DeviceMajor:0 DeviceMinor:246 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:261 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:266 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b66f962f7121d724fd502890519de74e9dbbf3005decd9e714b0a1c4bf68235d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b95482-e639-4d23-97a8-59941d9ecbe4/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cb8251a8-f48e-4af4-9606-5cbdea4a6b96/volumes/kubernetes.io~projected/kube-api-access-fdkxd DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:262 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8c3873471347ae6a64e3c7bb47fb8c52b1eac1cc9aa1073b118f3ddf84dbf11/userdata/shm DeviceMajor:0 DeviceMinor:329 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8273a1a4-5dbb-4f54-8f4c-4207cc11c228/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90383a53ed9782bf968aff14e6c06a34b754360b77ab2524c4a17df8fc66c857/userdata/shm DeviceMajor:0 DeviceMinor:268 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:264 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1bfbbe06579eaad0b3ea2580255b0879fa7ab032bc3a4994632eafc50d1686a/userdata/shm DeviceMajor:0 DeviceMinor:307 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7750339b-7420-4dda-904e-8221ba6c7ffe/volumes/kubernetes.io~projected/kube-api-access-mjzq9 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes/kubernetes.io~projected/kube-api-access-8qjx5 DeviceMajor:0 DeviceMinor:269 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b95482-e639-4d23-97a8-59941d9ecbe4/volumes/kubernetes.io~projected/kube-api-access-bdspl DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5ff45e7d5458dadc758ecfead2f4d8d6dfb94f8417418405e9e6928e059e64d/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:4a354f240e69f14 MacAddress:5e:2d:bb:45:b8:46 Speed:10000 Mtu:8901} {Name:6fc781808bb6125 MacAddress:ae:aa:ae:6e:58:51 Speed:10000 Mtu:8901} {Name:8182981a1498c38 MacAddress:5a:cb:df:db:42:2d Speed:10000 Mtu:8901} {Name:90383a53ed9782b MacAddress:ae:58:2d:19:ce:c7 Speed:10000 Mtu:8901} {Name:b1bfbbe06579eaa MacAddress:42:e5:d3:3e:86:3e Speed:10000 Mtu:8901} {Name:b8c3873471347ae MacAddress:ea:18:08:b0:31:a6 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d7:03:40:8b Speed:0 Mtu:9001} {Name:br-int MacAddress:fa:05:2f:cb:30:16 Speed:0 Mtu:8901} {Name:d5ff45e7d5458da MacAddress:a6:93:dd:62:30:f0 Speed:10000 Mtu:8901} {Name:dfad3869bcc60f8 MacAddress:22:d0:85:6a:f2:9d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d7:03:40:8b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:9a:94:b6:d2:5e:59 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b6:d7:d2:43:cc:5a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 19:27:23.804787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.804643    5237 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 19:27:23.804787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.804689    5237 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 19:27:23.805027 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.804970    5237 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 19:27:23.805122 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.804992    5237 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-29.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 19:27:23.805166 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.805124    5237 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 19:27:23.805166 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.805131    5237 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 19:27:23.805166 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.805141    5237 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:27:23.805166 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.805153    5237 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:27:23.805263 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.805235    5237 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 19:27:23.805424 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.805414    5237 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 19:27:23.805527 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.805518    5237 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 19:27:23.805563 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.805555    5237 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 19:27:23.805591 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.805571    5237 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 19:27:23.805591 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.805583    5237 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 19:27:23.805655 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.805592    5237 kubelet.go:405] "Adding apiserver pod source"
Nov 03 19:27:23.805655 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.805600    5237 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 19:27:23.807392 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.807076    5237 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 19:27:23.807392 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.807098    5237 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:27:23.808170 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.808146    5237 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 19:27:23.811567 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.811554    5237 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 19:27:23.811611 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.811568    5237 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:27:23.811680 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.811671    5237 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 19:27:23.811892 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.811880    5237 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 19:27:23.811943 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.811910    5237 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 19:27:23.812048 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.812036    5237 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 19:27:23.812078 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.812058    5237 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 19:27:23.812078 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.812068    5237 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 19:27:23.812128 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.812077    5237 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 19:27:23.812128 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.812086    5237 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 19:27:23.812128 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.812095    5237 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 19:27:23.812128 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.812112    5237 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 19:27:23.812128 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.812118    5237 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 19:27:23.812128 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.812125    5237 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 19:27:23.812285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.812131    5237 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 19:27:23.812285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.812152    5237 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 19:27:23.812285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.812165    5237 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 19:27:23.812285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.812187    5237 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 19:27:23.812285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.812192    5237 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 19:27:23.815673 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.815660    5237 server.go:1263] "Started kubelet"
Nov 03 19:27:23.815767 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.815735    5237 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 19:27:23.815893 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.815826    5237 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 19:27:23.815958 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.815938    5237 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 19:27:23.816242 ip-10-0-133-29 systemd[1]: Started Kubernetes Kubelet.
Nov 03 19:27:23.820241 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.816283    5237 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 19:27:23.820241 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.816928    5237 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 19:27:23.820241 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.819903    5237 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 19:27:23.820241 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.819971    5237 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 19:27:23.820766 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.820736    5237 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:21:14 +0000 UTC" deadline="2028-07-05 17:28:18.133735501 +0000 UTC"
Nov 03 19:27:23.820850 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.820768    5237 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14638h0m54.312971788s"
Nov 03 19:27:23.820850 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.820822    5237 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 19:27:23.820850 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.820848    5237 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 19:27:23.821116 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.821104    5237 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 19:27:23.822224 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:27:23.822198    5237 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-29.us-east-2.compute.internal\" not found"
Nov 03 19:27:23.824666 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.824644    5237 factory.go:55] Registering systemd factory
Nov 03 19:27:23.824666 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.824667    5237 factory.go:223] Registration of the systemd container factory successfully
Nov 03 19:27:23.825131 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.825095    5237 factory.go:153] Registering CRI-O factory
Nov 03 19:27:23.825131 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.825111    5237 factory.go:223] Registration of the crio container factory successfully
Nov 03 19:27:23.825251 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.825180    5237 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 19:27:23.825251 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.825207    5237 factory.go:103] Registering Raw factory
Nov 03 19:27:23.825251 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.825229    5237 manager.go:1199] Started watching for new ooms in manager
Nov 03 19:27:23.826099 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.825655    5237 manager.go:319] Starting recovery of all containers
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826050    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f6a7918-e7d4-495f-9514-82e883318c90" volumeName="kubernetes.io/projected/2f6a7918-e7d4-495f-9514-82e883318c90-kube-api-access-lthnm" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826295    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea78abe-c045-41df-b4c8-d5275dcb19e4" volumeName="kubernetes.io/configmap/4ea78abe-c045-41df-b4c8-d5275dcb19e4-config-volume" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826333    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb8251a8-f48e-4af4-9606-5cbdea4a6b96" volumeName="kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826360    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b95482-e639-4d23-97a8-59941d9ecbe4" volumeName="kubernetes.io/configmap/f9b95482-e639-4d23-97a8-59941d9ecbe4-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826380    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d79046e-eee4-4336-a852-131f013288a7" volumeName="kubernetes.io/projected/7d79046e-eee4-4336-a852-131f013288a7-kube-api-access-b45zq" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826441    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0827774e-9562-487f-9c6e-0934ef8707ea" volumeName="kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-tls-assets" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826459    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24476d57-ccbb-45a7-a594-5ba46b48ea41" volumeName="kubernetes.io/projected/24476d57-ccbb-45a7-a594-5ba46b48ea41-kube-api-access-twxzs" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826499    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea78abe-c045-41df-b4c8-d5275dcb19e4" volumeName="kubernetes.io/projected/4ea78abe-c045-41df-b4c8-d5275dcb19e4-kube-api-access-xlkr2" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826565    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65df60f4-e551-4351-a403-26305bece450" volumeName="kubernetes.io/empty-dir/65df60f4-e551-4351-a403-26305bece450-node-exporter-textfile" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826624    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d79046e-eee4-4336-a852-131f013288a7" volumeName="kubernetes.io/configmap/7d79046e-eee4-4336-a852-131f013288a7-iptables-alerter-script" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826658    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0efd5bb-5c99-4c84-a77f-d816995e6f79" volumeName="kubernetes.io/projected/f0efd5bb-5c99-4c84-a77f-d816995e6f79-kube-api-access-cvttl" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826690    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0827774e-9562-487f-9c6e-0934ef8707ea" volumeName="kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826715    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0827774e-9562-487f-9c6e-0934ef8707ea" volumeName="kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-config-volume" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826745    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74bbf640-fdad-4295-8810-f471ee4bb4da" volumeName="kubernetes.io/projected/74bbf640-fdad-4295-8810-f471ee4bb4da-kube-api-access-dmg4z" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826775    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7750339b-7420-4dda-904e-8221ba6c7ffe" volumeName="kubernetes.io/projected/7750339b-7420-4dda-904e-8221ba6c7ffe-kube-api-access-mjzq9" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826793    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b95482-e639-4d23-97a8-59941d9ecbe4" volumeName="kubernetes.io/projected/f9b95482-e639-4d23-97a8-59941d9ecbe4-kube-api-access-bdspl" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826850    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8273a1a4-5dbb-4f54-8f4c-4207cc11c228" volumeName="kubernetes.io/empty-dir/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-tmp" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826888    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8273a1a4-5dbb-4f54-8f4c-4207cc11c228" volumeName="kubernetes.io/projected/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-kube-api-access-grpgt" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826911    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9095ae71-1326-4ec3-9755-164738a4da17" volumeName="kubernetes.io/configmap/9095ae71-1326-4ec3-9755-164738a4da17-konnectivity-ca" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826949    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb8251a8-f48e-4af4-9606-5cbdea4a6b96" volumeName="kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826968    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b95482-e639-4d23-97a8-59941d9ecbe4" volumeName="kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-client-ca-bundle" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.826983    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9095ae71-1326-4ec3-9755-164738a4da17" volumeName="kubernetes.io/secret/9095ae71-1326-4ec3-9755-164738a4da17-agent-certs" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.827014    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b95482-e639-4d23-97a8-59941d9ecbe4" volumeName="kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-secret-metrics-server-tls" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.827068    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90836408-13df-4995-acec-b7bf40eeec84" volumeName="kubernetes.io/configmap/90836408-13df-4995-acec-b7bf40eeec84-serviceca" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.827104    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98cda745-c0e3-45ce-bcb0-5e99dea4f1e4" volumeName="kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovnkube-config" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.827164    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0827774e-9562-487f-9c6e-0934ef8707ea" volumeName="kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.827183    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0827774e-9562-487f-9c6e-0934ef8707ea" volumeName="kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.827204    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74bbf640-fdad-4295-8810-f471ee4bb4da" volumeName="kubernetes.io/empty-dir/74bbf640-fdad-4295-8810-f471ee4bb4da-tmp-dir" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.827274    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b22e16a-511b-4d70-a39c-16160ef790b1" volumeName="kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.827304    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90836408-13df-4995-acec-b7bf40eeec84" volumeName="kubernetes.io/projected/90836408-13df-4995-acec-b7bf40eeec84-kube-api-access-v9fmg" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.827323    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b95482-e639-4d23-97a8-59941d9ecbe4" volumeName="kubernetes.io/configmap/f9b95482-e639-4d23-97a8-59941d9ecbe4-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.827351    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0827774e-9562-487f-9c6e-0934ef8707ea" volumeName="kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:27:23.827624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.827384    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0827774e-9562-487f-9c6e-0934ef8707ea" volumeName="kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 19:27:23.829124 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.827449    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f6a7918-e7d4-495f-9514-82e883318c90" volumeName="kubernetes.io/empty-dir/2f6a7918-e7d4-495f-9514-82e883318c90-data-volume" seLinuxMountContext=""
Nov 03 19:27:23.829226 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829141    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65df60f4-e551-4351-a403-26305bece450" volumeName="kubernetes.io/configmap/65df60f4-e551-4351-a403-26305bece450-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 19:27:23.829226 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829180    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98108a97-4bce-4291-8c25-ef6a88d542e4" volumeName="kubernetes.io/projected/98108a97-4bce-4291-8c25-ef6a88d542e4-kube-api-access-np75z" seLinuxMountContext=""
Nov 03 19:27:23.829323 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829195    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24476d57-ccbb-45a7-a594-5ba46b48ea41" volumeName="kubernetes.io/secret/24476d57-ccbb-45a7-a594-5ba46b48ea41-cert" seLinuxMountContext=""
Nov 03 19:27:23.829323 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829269    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65df60f4-e551-4351-a403-26305bece450" volumeName="kubernetes.io/configmap/65df60f4-e551-4351-a403-26305bece450-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:27:23.829323 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829283    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65df60f4-e551-4351-a403-26305bece450" volumeName="kubernetes.io/secret/65df60f4-e551-4351-a403-26305bece450-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 19:27:23.829452 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829325    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98cda745-c0e3-45ce-bcb0-5e99dea4f1e4" volumeName="kubernetes.io/secret/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 19:27:23.829452 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829339    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98cda745-c0e3-45ce-bcb0-5e99dea4f1e4" volumeName="kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-env-overrides" seLinuxMountContext=""
Nov 03 19:27:23.829452 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829353    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98cda745-c0e3-45ce-bcb0-5e99dea4f1e4" volumeName="kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 19:27:23.829452 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829365    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0827774e-9562-487f-9c6e-0934ef8707ea" volumeName="kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-kube-api-access-8qjx5" seLinuxMountContext=""
Nov 03 19:27:23.829452 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829380    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea78abe-c045-41df-b4c8-d5275dcb19e4" volumeName="kubernetes.io/empty-dir/4ea78abe-c045-41df-b4c8-d5275dcb19e4-tmp-dir" seLinuxMountContext=""
Nov 03 19:27:23.829452 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829407    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea78abe-c045-41df-b4c8-d5275dcb19e4" volumeName="kubernetes.io/secret/4ea78abe-c045-41df-b4c8-d5275dcb19e4-metrics-tls" seLinuxMountContext=""
Nov 03 19:27:23.829452 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829422    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65df60f4-e551-4351-a403-26305bece450" volumeName="kubernetes.io/secret/65df60f4-e551-4351-a403-26305bece450-node-exporter-tls" seLinuxMountContext=""
Nov 03 19:27:23.829452 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829434    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0827774e-9562-487f-9c6e-0934ef8707ea" volumeName="kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-main-db" seLinuxMountContext=""
Nov 03 19:27:23.829452 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829450    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0827774e-9562-487f-9c6e-0934ef8707ea" volumeName="kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-config-out" seLinuxMountContext=""
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829481    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0827774e-9562-487f-9c6e-0934ef8707ea" volumeName="kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-cluster-tls-config" seLinuxMountContext=""
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829502    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98108a97-4bce-4291-8c25-ef6a88d542e4" volumeName="kubernetes.io/configmap/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-daemon-config" seLinuxMountContext=""
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829517    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aef23c0d-d927-4a6e-bc22-288027b498a8" volumeName="kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729" seLinuxMountContext=""
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829530    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb8251a8-f48e-4af4-9606-5cbdea4a6b96" volumeName="kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cni-binary-copy" seLinuxMountContext=""
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829564    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f6a7918-e7d4-495f-9514-82e883318c90" volumeName="kubernetes.io/secret/2f6a7918-e7d4-495f-9514-82e883318c90-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829573    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8273a1a4-5dbb-4f54-8f4c-4207cc11c228" volumeName="kubernetes.io/empty-dir/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-tuned" seLinuxMountContext=""
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829581    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98cda745-c0e3-45ce-bcb0-5e99dea4f1e4" volumeName="kubernetes.io/projected/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-kube-api-access-r8hvm" seLinuxMountContext=""
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829591    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7750339b-7420-4dda-904e-8221ba6c7ffe" volumeName="kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs" seLinuxMountContext=""
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829651    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0827774e-9562-487f-9c6e-0934ef8707ea" volumeName="kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829668    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98108a97-4bce-4291-8c25-ef6a88d542e4" volumeName="kubernetes.io/configmap/98108a97-4bce-4291-8c25-ef6a88d542e4-cni-binary-copy" seLinuxMountContext=""
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829680    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b95482-e639-4d23-97a8-59941d9ecbe4" volumeName="kubernetes.io/empty-dir/f9b95482-e639-4d23-97a8-59941d9ecbe4-audit-log" seLinuxMountContext=""
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829693    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65df60f4-e551-4351-a403-26305bece450" volumeName="kubernetes.io/projected/65df60f4-e551-4351-a403-26305bece450-kube-api-access-ql6s2" seLinuxMountContext=""
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829771    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb8251a8-f48e-4af4-9606-5cbdea4a6b96" volumeName="kubernetes.io/projected/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-kube-api-access-fdkxd" seLinuxMountContext=""
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829786    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b95482-e639-4d23-97a8-59941d9ecbe4" volumeName="kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829798    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0827774e-9562-487f-9c6e-0934ef8707ea" volumeName="kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-web-config" seLinuxMountContext=""
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829821    5237 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f6a7918-e7d4-495f-9514-82e883318c90" volumeName="kubernetes.io/configmap/2f6a7918-e7d4-495f-9514-82e883318c90-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829853    5237 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 19:27:23.829889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.829861    5237 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 19:27:23.830955 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.830934    5237 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:27:23.831183 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.831167    5237 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:27:23.832762 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.832738    5237 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:27:23.834980 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.834966    5237 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 19:27:23.850273 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.850228    5237 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 19:27:23.852165 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.852143    5237 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 19:27:23.852165 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.852163    5237 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 19:27:23.852165 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.852183    5237 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 19:27:23.852165 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:27:23.852220    5237 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 19:27:23.856589 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.856568    5237 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:27:23.861928 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.861892    5237 generic.go:356] "Generic (PLEG): container finished" podID="65df60f4-e551-4351-a403-26305bece450" containerID="bf3956da4a274d1de32a562d0ee431df2c08a7da8c2d3509b1d9b4ed3a1899c7" exitCode=0
Nov 03 19:27:23.867892 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.867864    5237 generic.go:356] "Generic (PLEG): container finished" podID="715d00a389ecabd279054a6ca23fc9ac" containerID="d0c5364d8ec8d2c3737218feb52caa79ff3e361cb6effbde7328099d809eacc2" exitCode=0
Nov 03 19:27:23.886405 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.886377    5237 generic.go:356] "Generic (PLEG): container finished" podID="cb8251a8-f48e-4af4-9606-5cbdea4a6b96" containerID="5c94f271c8cfb62fa97befdf82c7990eeb4232287d75c5b5c89da0eab04458ae" exitCode=0
Nov 03 19:27:23.886405 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.886403    5237 generic.go:356] "Generic (PLEG): container finished" podID="cb8251a8-f48e-4af4-9606-5cbdea4a6b96" containerID="069d675080f58a4373ae69c56bc2ee0568255d72df3c2ddcf068ae46ba8aec9d" exitCode=0
Nov 03 19:27:23.886405 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.886410    5237 generic.go:356] "Generic (PLEG): container finished" podID="cb8251a8-f48e-4af4-9606-5cbdea4a6b96" containerID="ccc3e5b69a4cef78aaf623226e596d58981c7ef821095a878ee8b5f5ea46a4a4" exitCode=0
Nov 03 19:27:23.886600 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.886416    5237 generic.go:356] "Generic (PLEG): container finished" podID="cb8251a8-f48e-4af4-9606-5cbdea4a6b96" containerID="da56acb2d3195eef5e218bff82876a2408faf44e2073f177675a3ef7bba45e60" exitCode=0
Nov 03 19:27:23.886600 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.886422    5237 generic.go:356] "Generic (PLEG): container finished" podID="cb8251a8-f48e-4af4-9606-5cbdea4a6b96" containerID="3e4987959f519a673b4f37360768705d0d07aecd81e00d5927ff05fa9d616de9" exitCode=0
Nov 03 19:27:23.886600 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.886428    5237 generic.go:356] "Generic (PLEG): container finished" podID="cb8251a8-f48e-4af4-9606-5cbdea4a6b96" containerID="76c4fa47a463ab9241093ceb9f604556117e574983b4f18ccc5de032bc44936d" exitCode=0
Nov 03 19:27:23.890082 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.890052    5237 generic.go:356] "Generic (PLEG): container finished" podID="0827774e-9562-487f-9c6e-0934ef8707ea" containerID="dcc926ed375a62f53970a52e5d65852dbdcb9cda977e99db05668588f549c204" exitCode=0
Nov 03 19:27:23.919826 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.919789    5237 manager.go:324] Recovery completed
Nov 03 19:27:23.938940 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.938867    5237 cpu_manager.go:226] "Starting" policy="none"
Nov 03 19:27:23.938940 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.938886    5237 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 19:27:23.938940 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.938908    5237 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 19:27:23.939127 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.939029    5237 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 19:27:23.939127 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.939040    5237 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 19:27:23.939127 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.939057    5237 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 19:27:23.939127 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.939063    5237 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 19:27:23.939127 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.939068    5237 policy_none.go:50] "Start"
Nov 03 19:27:23.939127 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.939076    5237 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 19:27:23.939127 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.939087    5237 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 19:27:23.940038 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.940024    5237 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 19:27:23.940071 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.940038    5237 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 19:27:23.940071 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.940044    5237 policy_none.go:44] "Start"
Nov 03 19:27:23.944301 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.944275    5237 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 19:27:23.946816 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.946804    5237 manager.go:341] "Starting Device Plugin manager"
Nov 03 19:27:23.947599 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:27:23.947575    5237 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 19:27:23.947692 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.947600    5237 server.go:87] "Starting device plugin registration server"
Nov 03 19:27:23.948119 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.948086    5237 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 19:27:23.948119 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.948106    5237 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 19:27:23.948265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.948190    5237 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 19:27:23.948326 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.948284    5237 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 19:27:23.948326 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.948321    5237 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 19:27:23.948407 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.948328    5237 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 19:27:23.948463 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.948382    5237 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:27:23.94830977Z","UUID":"6241d1b1-bf01-461c-a6d8-192bb93e077f","Handler":null,"Name":"","Endpoint":""}
Nov 03 19:27:23.949628 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.949607    5237 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 19:27:23.949736 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.949637    5237 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 19:27:23.950887 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:27:23.950859    5237 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 19:27:23.952373 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.952332    5237 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal"]
Nov 03 19:27:23.953269 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.953246    5237 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:23.953378 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.953242    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal" event={"ID":"715d00a389ecabd279054a6ca23fc9ac","Type":"ContainerStarted","Data":"9541278ef73e638d1920a6e8ba31e6ddaf76d01439d9f6b27d1263b8b0e1827a"}
Nov 03 19:27:23.953378 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.953306    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal" event={"ID":"715d00a389ecabd279054a6ca23fc9ac","Type":"ContainerDied","Data":"d0c5364d8ec8d2c3737218feb52caa79ff3e361cb6effbde7328099d809eacc2"}
Nov 03 19:27:23.953378 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.953327    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal" event={"ID":"715d00a389ecabd279054a6ca23fc9ac","Type":"ContainerStarted","Data":"ed1e72c10959b9f9352dd0126173841cdfa6acce1eebd375ed73143110000e55"}
Nov 03 19:27:23.953533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.953429    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal" event={"ID":"dd9e417c99cd1e2d8fd47e9d585da591","Type":"ContainerStarted","Data":"244fe5d5b0655def75d29c33708816f58eb1936c6bc88ea714e689eb2cd893d0"}
Nov 03 19:27:23.953533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.953445    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal" event={"ID":"dd9e417c99cd1e2d8fd47e9d585da591","Type":"ContainerStarted","Data":"5c6dedb710a0f80631462172893a46d7295f769277996791e99c4a9614d0998e"}
Nov 03 19:27:23.953691 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.953668    5237 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:23.973294 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.973267    5237 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:27:23.973414 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:27:23.973313    5237 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:23.973414 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:23.973325    5237 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:27:23.973414 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:27:23.973378    5237 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:24.035137 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.035107    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/715d00a389ecabd279054a6ca23fc9ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"715d00a389ecabd279054a6ca23fc9ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:24.035137 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.035138    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/715d00a389ecabd279054a6ca23fc9ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"715d00a389ecabd279054a6ca23fc9ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:24.035319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.035156    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd9e417c99cd1e2d8fd47e9d585da591-config\") pod \"kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"dd9e417c99cd1e2d8fd47e9d585da591\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:24.048800 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.048775    5237 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:27:24.050846 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.050817    5237 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:27:24.050931 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.050875    5237 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:27:24.050931 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.050889    5237 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-29.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:27:24.050931 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.050928    5237 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:24.073387 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.073359    5237 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:24.073507 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.073436    5237 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:24.135474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.135436    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/715d00a389ecabd279054a6ca23fc9ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"715d00a389ecabd279054a6ca23fc9ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:24.135474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.135450    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/715d00a389ecabd279054a6ca23fc9ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"715d00a389ecabd279054a6ca23fc9ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:24.135677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.135502    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/715d00a389ecabd279054a6ca23fc9ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"715d00a389ecabd279054a6ca23fc9ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:24.135677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.135522    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd9e417c99cd1e2d8fd47e9d585da591-config\") pod \"kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"dd9e417c99cd1e2d8fd47e9d585da591\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:24.135677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.135550    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd9e417c99cd1e2d8fd47e9d585da591-config\") pod \"kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"dd9e417c99cd1e2d8fd47e9d585da591\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:24.135677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.135606    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/715d00a389ecabd279054a6ca23fc9ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal\" (UID: \"715d00a389ecabd279054a6ca23fc9ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:24.806570 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.806528    5237 apiserver.go:52] "Watching apiserver"
Nov 03 19:27:24.815668 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.815647    5237 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 19:27:24.816326 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.816302    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h6nf4","kube-system/konnectivity-agent-52ft7","openshift-monitoring/node-exporter-z24qr","openshift-monitoring/alertmanager-main-0","kube-system/kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal","openshift-ingress-canary/ingress-canary-xszjb","openshift-monitoring/metrics-server-7d8d7db6bc-592nt","openshift-image-registry/node-ca-2n8jc","openshift-multus/network-metrics-daemon-tz6b2","openshift-dns/node-resolver-r2j57","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-494mb","openshift-multus/multus-additional-cni-plugins-s5hhz","openshift-network-operator/iptables-alerter-sp6pz","openshift-insights/insights-runtime-extractor-hkjp4","openshift-multus/multus-pc8t6","kube-system/global-pull-secret-syncer-lqnwg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5","openshift-ovn-kubernetes/ovnkube-node-fqmk8","openshift-cluster-node-tuning-operator/tuned-6ksh4"]
Nov 03 19:27:24.823175 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823149    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 19:27:24.823307 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823214    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 19:27:24.823307 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823226    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 19:27:24.823307 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823228    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 19:27:24.823307 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823255    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 19:27:24.823307 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823289    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 19:27:24.823307 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823297    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 19:27:24.823593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823343    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g6kk8\""
Nov 03 19:27:24.823593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823369    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 19:27:24.823593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823403    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lsdlf\""
Nov 03 19:27:24.823593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823448    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 19:27:24.823593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823478    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 19:27:24.823593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823484    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s8pjd\""
Nov 03 19:27:24.823593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823492    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:27:24.823593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823213    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 19:27:24.823593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823595    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 19:27:24.824226 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823639    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 19:27:24.824226 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.823694    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 19:27:24.824226 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.824007    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f8cx9\""
Nov 03 19:27:24.824226 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.824078    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 19:27:24.824226 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.824144    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5vlcb\""
Nov 03 19:27:24.824453 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.824321    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-st58b\""
Nov 03 19:27:24.824453 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.824400    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 19:27:24.824547 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.824462    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:27:24.824547 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.824528    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 19:27:24.826949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.826920    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:27:24.826949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.826932    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lmsjd\""
Nov 03 19:27:24.826949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.826941    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 19:27:24.827106 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.826943    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 19:27:24.827106 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.826979    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 19:27:24.827106 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.826986    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 19:27:24.827106 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.827022    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xv5d7\""
Nov 03 19:27:24.827669 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.827634    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 19:27:24.827797 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.827778    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:27:24.827951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.827936    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-hwgf7\""
Nov 03 19:27:24.828030 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.828017    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r6b5k\""
Nov 03 19:27:24.828096 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.828081    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xp5ps\""
Nov 03 19:27:24.828739 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.828725    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6b2s9\""
Nov 03 19:27:24.828774 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.828736    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xlp4b\""
Nov 03 19:27:24.828808 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.828779    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 19:27:24.829483 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.829466    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 19:27:24.829553 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.829496    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 19:27:24.829631 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.829620    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 19:27:24.829663 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.829656    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 19:27:24.829692 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.829677    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 19:27:24.830418 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.830406    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:27:24.830470 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.830420    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:27:24.831101 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831080    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 19:27:24.831176 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831095    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gtgvx\""
Nov 03 19:27:24.831176 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831079    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 19:27:24.831176 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831114    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 19:27:24.831312 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831167    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 19:27:24.831312 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831123    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 19:27:24.831312 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831117    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 19:27:24.831312 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831212    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 19:27:24.831312 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831216    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 19:27:24.831542 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831433    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 19:27:24.831618 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831598    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 19:27:24.831715 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831643    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 19:27:24.831715 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831657    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 19:27:24.831715 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831664    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-qhgjf\""
Nov 03 19:27:24.831715 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831670    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 19:27:24.831715 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831648    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 19:27:24.831715 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831712    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6lz2t\""
Nov 03 19:27:24.831972 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831733    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 19:27:24.831972 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.831966    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 19:27:24.832405 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.832391    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 19:27:24.832455 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.832411    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 19:27:24.832488 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.832477    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 19:27:24.833050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.833032    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kw62n\""
Nov 03 19:27:24.833133 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.833071    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 19:27:24.833213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.833187    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 19:27:24.833213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.833208    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 19:27:24.834219 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.834203    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2720n911htvu6\""
Nov 03 19:27:24.834264 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.834242    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4sj62\""
Nov 03 19:27:24.834302 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.834265    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 19:27:24.834302 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.834279    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 19:27:24.836300 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.836284    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 19:27:24.839616 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.839601    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-cni-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.839669 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.839631    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-cnibin\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.839669 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.839649    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-daemon-config\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.839669 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.839665    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovnkube-script-lib\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.839772 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.839717    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-device-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.839772 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.839740    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:27:24.839772 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.839762    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-lib-modules\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.839916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.839785    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-var-lib-kubelet\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.839916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.839820    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-hostroot\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.840023 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.839937    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-daemon-config\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.840023 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.839982    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-log-socket\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.840023 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.840007    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-socket-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.840173 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.840049    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-run\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.840173 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.840080    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grpgt\" (UniqueName: \"kubernetes.io/projected/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-kube-api-access-grpgt\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.840173 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.840130    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:27:24.840173 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.840155    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovnkube-script-lib\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.840358 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.840187    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-conf-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.840358 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.840216    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-etc-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.840358 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.840220    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:27:24.840358 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.840305    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmg4z\" (UniqueName: \"kubernetes.io/projected/74bbf640-fdad-4295-8810-f471ee4bb4da-kube-api-access-dmg4z\") pod \"node-resolver-r2j57\" (UID: \"74bbf640-fdad-4295-8810-f471ee4bb4da\") " pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:27:24.840358 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.840341    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-system-cni-dir\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:27:24.840527 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.840378    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cnibin\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:27:24.840907 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.840452    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-modprobe-d\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.840987 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.840956    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-k8s-cni-cncf-io\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.841042 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.841004    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-cni-bin\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.841042 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.841019    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b22e16a-511b-4d70-a39c-16160ef790b1-original-pull-secret\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:27:24.841129 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.841056    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-registration-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.841272 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.841250    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-sys-fs\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.841395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.841287    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9095ae71-1326-4ec3-9755-164738a4da17-konnectivity-ca\") pod \"konnectivity-agent-52ft7\" (UID: \"9095ae71-1326-4ec3-9755-164738a4da17\") " pod="kube-system/konnectivity-agent-52ft7"
Nov 03 19:27:24.841395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.841324    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-cni-multus\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.841395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.841355    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-systemd\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.841395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.841386    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-var-lib-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.841619 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.841566    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.841674 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.841618    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.841800 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.841773    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74bbf640-fdad-4295-8810-f471ee4bb4da-hosts-file\") pod \"node-resolver-r2j57\" (UID: \"74bbf640-fdad-4295-8810-f471ee4bb4da\") " pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:27:24.841938 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.841918    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-netns\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.841938 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.841920    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9095ae71-1326-4ec3-9755-164738a4da17-konnectivity-ca\") pod \"konnectivity-agent-52ft7\" (UID: \"9095ae71-1326-4ec3-9755-164738a4da17\") " pod="kube-system/konnectivity-agent-52ft7"
Nov 03 19:27:24.842038 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842001    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-run-netns\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.842097 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842038    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-cni-netd\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.842097 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842085    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8hvm\" (UniqueName: \"kubernetes.io/projected/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-kube-api-access-r8hvm\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.842181 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842119    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvttl\" (UniqueName: \"kubernetes.io/projected/f0efd5bb-5c99-4c84-a77f-d816995e6f79-kube-api-access-cvttl\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.842181 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842145    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90836408-13df-4995-acec-b7bf40eeec84-host\") pod \"node-ca-2n8jc\" (UID: \"90836408-13df-4995-acec-b7bf40eeec84\") " pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:27:24.842181 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842175    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-kubernetes\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.842310 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842259    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-tmp\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.842310 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842284    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-system-cni-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.842392 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842315    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-cni-bin\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.842392 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842348    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np75z\" (UniqueName: \"kubernetes.io/projected/98108a97-4bce-4291-8c25-ef6a88d542e4-kube-api-access-np75z\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.842392 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842382    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.842516 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842413    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9095ae71-1326-4ec3-9755-164738a4da17-agent-certs\") pod \"konnectivity-agent-52ft7\" (UID: \"9095ae71-1326-4ec3-9755-164738a4da17\") " pod="kube-system/konnectivity-agent-52ft7"
Nov 03 19:27:24.842516 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842437    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-slash\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.842516 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842468    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/90836408-13df-4995-acec-b7bf40eeec84-serviceca\") pod \"node-ca-2n8jc\" (UID: \"90836408-13df-4995-acec-b7bf40eeec84\") " pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:27:24.842652 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842629    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9fmg\" (UniqueName: \"kubernetes.io/projected/90836408-13df-4995-acec-b7bf40eeec84-kube-api-access-v9fmg\") pod \"node-ca-2n8jc\" (UID: \"90836408-13df-4995-acec-b7bf40eeec84\") " pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:27:24.842701 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842674    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdkxd\" (UniqueName: \"kubernetes.io/projected/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-kube-api-access-fdkxd\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:27:24.842748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842708    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysconfig\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.842748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842737    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b22e16a-511b-4d70-a39c-16160ef790b1-kubelet-config\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:27:24.842830 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842762    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b22e16a-511b-4d70-a39c-16160ef790b1-dbus\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:27:24.842830 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842791    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98108a97-4bce-4291-8c25-ef6a88d542e4-cni-binary-copy\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.842830 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842829    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjzq9\" (UniqueName: \"kubernetes.io/projected/7750339b-7420-4dda-904e-8221ba6c7ffe-kube-api-access-mjzq9\") pod \"network-metrics-daemon-tz6b2\" (UID: \"7750339b-7420-4dda-904e-8221ba6c7ffe\") " pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:27:24.842994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842876    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovnkube-config\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.842994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842901    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74bbf640-fdad-4295-8810-f471ee4bb4da-tmp-dir\") pod \"node-resolver-r2j57\" (UID: \"74bbf640-fdad-4295-8810-f471ee4bb4da\") " pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:27:24.842994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842931    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-systemd\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.842994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842960    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-tuned\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.842994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.842990    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:27:24.843206 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.843021    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cni-binary-copy\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:27:24.843206 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.843047    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysctl-d\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.843206 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.843109    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-sys\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.843206 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.843136    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs\") pod \"network-metrics-daemon-tz6b2\" (UID: \"7750339b-7420-4dda-904e-8221ba6c7ffe\") " pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:27:24.843805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.843783    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovn-node-metrics-cert\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.843909 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.843829    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-socket-dir-parent\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.843909 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.843890    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-kubelet\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.844005 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.843938    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-multus-certs\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.844005 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.843972    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-env-overrides\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.844096 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.844016    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:27:24.844096 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.844050    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysctl-conf\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.844182 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.844088    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-host\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.844182 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.844118    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.844182 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.844146    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-ovn\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.844310 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.844187    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-node-log\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.844310 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.844219    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-etc-selinux\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.844310 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.844288    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-etc-kubernetes\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.844427 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.844332    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-kubelet\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.844427 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.844363    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-systemd-units\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.844996 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.844933    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74bbf640-fdad-4295-8810-f471ee4bb4da-tmp-dir\") pod \"node-resolver-r2j57\" (UID: \"74bbf640-fdad-4295-8810-f471ee4bb4da\") " pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:27:24.845489 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.845467    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-tuned\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.845599 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.845303    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7750339b-7420-4dda-904e-8221ba6c7ffe-metrics-certs\") pod \"network-metrics-daemon-tz6b2\" (UID: \"7750339b-7420-4dda-904e-8221ba6c7ffe\") " pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:27:24.845684 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.844818    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-tmp\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.845861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.845499    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:27:24.845861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.845514    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9095ae71-1326-4ec3-9755-164738a4da17-agent-certs\") pod \"konnectivity-agent-52ft7\" (UID: \"9095ae71-1326-4ec3-9755-164738a4da17\") " pod="kube-system/konnectivity-agent-52ft7"
Nov 03 19:27:24.845989 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.845594    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-env-overrides\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.845989 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.845701    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cni-binary-copy\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:27:24.845989 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.845702    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98108a97-4bce-4291-8c25-ef6a88d542e4-cni-binary-copy\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.845989 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.845725    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovn-node-metrics-cert\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.845989 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.845920    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-ovnkube-config\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.845989 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.845932    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/90836408-13df-4995-acec-b7bf40eeec84-serviceca\") pod \"node-ca-2n8jc\" (UID: \"90836408-13df-4995-acec-b7bf40eeec84\") " pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:27:24.850688 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.850669    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grpgt\" (UniqueName: \"kubernetes.io/projected/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-kube-api-access-grpgt\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.854609 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.854592    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmg4z\" (UniqueName: \"kubernetes.io/projected/74bbf640-fdad-4295-8810-f471ee4bb4da-kube-api-access-dmg4z\") pod \"node-resolver-r2j57\" (UID: \"74bbf640-fdad-4295-8810-f471ee4bb4da\") " pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:27:24.858012 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.857992    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjzq9\" (UniqueName: \"kubernetes.io/projected/7750339b-7420-4dda-904e-8221ba6c7ffe-kube-api-access-mjzq9\") pod \"network-metrics-daemon-tz6b2\" (UID: \"7750339b-7420-4dda-904e-8221ba6c7ffe\") " pod="openshift-multus/network-metrics-daemon-tz6b2"
Nov 03 19:27:24.858129 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.858108    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np75z\" (UniqueName: \"kubernetes.io/projected/98108a97-4bce-4291-8c25-ef6a88d542e4-kube-api-access-np75z\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.858190 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.858153    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdkxd\" (UniqueName: \"kubernetes.io/projected/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-kube-api-access-fdkxd\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:27:24.858190 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.858177    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9fmg\" (UniqueName: \"kubernetes.io/projected/90836408-13df-4995-acec-b7bf40eeec84-kube-api-access-v9fmg\") pod \"node-ca-2n8jc\" (UID: \"90836408-13df-4995-acec-b7bf40eeec84\") " pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:27:24.858292 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.858239    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvttl\" (UniqueName: \"kubernetes.io/projected/f0efd5bb-5c99-4c84-a77f-d816995e6f79-kube-api-access-cvttl\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.858397 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.858382    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8hvm\" (UniqueName: \"kubernetes.io/projected/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-kube-api-access-r8hvm\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.903466 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.903440    5237 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:24.918278 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.918252    5237 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:27:24.918392 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:27:24.918293    5237 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.us-east-2.compute.internal"
Nov 03 19:27:24.922618 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.922597    5237 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 19:27:24.945365 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945333    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65df60f4-e551-4351-a403-26305bece450-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:24.945496 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945380    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-device-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.945496 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945408    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-lib-modules\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.945496 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945431    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-var-lib-kubelet\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.945496 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945455    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-hostroot\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.945496 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945480    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-log-socket\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.945694 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945518    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b95482-e639-4d23-97a8-59941d9ecbe4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:24.945694 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945546    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-lib-modules\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.945694 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945562    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-var-lib-kubelet\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.945694 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945552    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-root\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:24.945694 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945548    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-log-socket\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.945694 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945590    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-hostroot\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.945969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945695    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-device-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.945969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945748    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65df60f4-e551-4351-a403-26305bece450-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:24.945969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945768    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-run\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.945969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945809    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-conf-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.945969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945855    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-etc-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.945969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945868    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-run\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.945969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945881    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-conf-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.945969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945882    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-secret-metrics-server-tls\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:24.945969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945920    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.945969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945930    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-etc-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.945969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945880    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b95482-e639-4d23-97a8-59941d9ecbe4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:24.945969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945949    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f6a7918-e7d4-495f-9514-82e883318c90-data-volume\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.945984    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-system-cni-dir\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946008    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-modprobe-d\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946032    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-sys-fs\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946046    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f6a7918-e7d4-495f-9514-82e883318c90-data-volume\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946060    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f9b95482-e639-4d23-97a8-59941d9ecbe4-metrics-server-audit-profiles\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946085    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/65df60f4-e551-4351-a403-26305bece450-node-exporter-tls\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946074    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-system-cni-dir\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946125    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946172    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946182    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946199    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74bbf640-fdad-4295-8810-f471ee4bb4da-hosts-file\") pod \"node-resolver-r2j57\" (UID: \"74bbf640-fdad-4295-8810-f471ee4bb4da\") " pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946210    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-modprobe-d\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946226    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-sys-fs\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946120    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-secret-metrics-server-tls\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946228    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65df60f4-e551-4351-a403-26305bece450-metrics-client-ca\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946275    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946306    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946330    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74bbf640-fdad-4295-8810-f471ee4bb4da-hosts-file\") pod \"node-resolver-r2j57\" (UID: \"74bbf640-fdad-4295-8810-f471ee4bb4da\") " pod="openshift-dns/node-resolver-r2j57"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946363    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946423    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f9b95482-e639-4d23-97a8-59941d9ecbe4-metrics-server-audit-profiles\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946502    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/65df60f4-e551-4351-a403-26305bece450-node-exporter-tls\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946524    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946524    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65df60f4-e551-4351-a403-26305bece450-metrics-client-ca\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:24.946533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946548    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-run-netns\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946577    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-cni-bin\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946601    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f6a7918-e7d4-495f-9514-82e883318c90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946607    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-run-netns\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946624    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-slash\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946632    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-cni-bin\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946655    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d79046e-eee4-4336-a852-131f013288a7-host-slash\") pod \"iptables-alerter-sp6pz\" (UID: \"7d79046e-eee4-4336-a852-131f013288a7\") " pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946689    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-slash\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946680    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea78abe-c045-41df-b4c8-d5275dcb19e4-config-volume\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946749    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlkr2\" (UniqueName: \"kubernetes.io/projected/4ea78abe-c045-41df-b4c8-d5275dcb19e4-kube-api-access-xlkr2\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946781    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f6a7918-e7d4-495f-9514-82e883318c90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946778    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946818    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl729\" (UniqueName: \"kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729\") pod \"network-check-target-494mb\" (UID: \"aef23c0d-d927-4a6e-bc22-288027b498a8\") " pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946868    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea78abe-c045-41df-b4c8-d5275dcb19e4-tmp-dir\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946891    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/65df60f4-e551-4351-a403-26305bece450-node-exporter-textfile\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946915    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b22e16a-511b-4d70-a39c-16160ef790b1-dbus\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946921    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea78abe-c045-41df-b4c8-d5275dcb19e4-config-volume\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946941    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f9b95482-e639-4d23-97a8-59941d9ecbe4-audit-log\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946966    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2f6a7918-e7d4-495f-9514-82e883318c90-crio-socket\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947012    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/65df60f4-e551-4351-a403-26305bece450-node-exporter-textfile\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.946992    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947011    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea78abe-c045-41df-b4c8-d5275dcb19e4-tmp-dir\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947045    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-secret-metrics-server-client-certs\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947081    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f9b95482-e639-4d23-97a8-59941d9ecbe4-audit-log\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947082    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdspl\" (UniqueName: \"kubernetes.io/projected/f9b95482-e639-4d23-97a8-59941d9ecbe4-kube-api-access-bdspl\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947144    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b22e16a-511b-4d70-a39c-16160ef790b1-dbus\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947182    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-sys\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947191    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947213    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6s2\" (UniqueName: \"kubernetes.io/projected/65df60f4-e551-4351-a403-26305bece450-kube-api-access-ql6s2\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947271    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-multus-certs\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947284    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947319    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-client-ca-bundle\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947338    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-multus-certs\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947350    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d79046e-eee4-4336-a852-131f013288a7-iptables-alerter-script\") pod \"iptables-alerter-sp6pz\" (UID: \"7d79046e-eee4-4336-a852-131f013288a7\") " pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947344    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-sys\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947379    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/65df60f4-e551-4351-a403-26305bece450-node-exporter-accelerators-collector-config\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947410    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysctl-conf\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947435    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f6a7918-e7d4-495f-9514-82e883318c90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947458    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-etc-kubernetes\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947466    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-secret-metrics-server-client-certs\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947488    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-kubelet\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947514    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-systemd-units\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947546    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b45zq\" (UniqueName: \"kubernetes.io/projected/7d79046e-eee4-4336-a852-131f013288a7-kube-api-access-b45zq\") pod \"iptables-alerter-sp6pz\" (UID: \"7d79046e-eee4-4336-a852-131f013288a7\") " pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947570    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ea78abe-c045-41df-b4c8-d5275dcb19e4-metrics-tls\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947585    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d79046e-eee4-4336-a852-131f013288a7-iptables-alerter-script\") pod \"iptables-alerter-sp6pz\" (UID: \"7d79046e-eee4-4336-a852-131f013288a7\") " pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:27:24.947610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947581    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f9b95482-e639-4d23-97a8-59941d9ecbe4-client-ca-bundle\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947649    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-etc-kubernetes\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947657    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-cni-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947593    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-cni-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947657    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysctl-conf\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947669    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/65df60f4-e551-4351-a403-26305bece450-node-exporter-accelerators-collector-config\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947695    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-cnibin\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947723    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-kubelet\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947732    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-socket-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947746    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-systemd-units\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947764    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twxzs\" (UniqueName: \"kubernetes.io/projected/24476d57-ccbb-45a7-a594-5ba46b48ea41-kube-api-access-twxzs\") pod \"ingress-canary-xszjb\" (UID: \"24476d57-ccbb-45a7-a594-5ba46b48ea41\") " pod="openshift-ingress-canary/ingress-canary-xszjb"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947802    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-cnibin\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947807    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947814    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-socket-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947864    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cnibin\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947893    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-k8s-cni-cncf-io\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947917    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ea78abe-c045-41df-b4c8-d5275dcb19e4-metrics-tls\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947919    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-cni-bin\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947927    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-cnibin\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947945    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-registration-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947968    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-web-config\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947971    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-k8s-cni-cncf-io\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947971    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f6a7918-e7d4-495f-9514-82e883318c90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947976    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-cni-bin\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.947993    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lthnm\" (UniqueName: \"kubernetes.io/projected/2f6a7918-e7d4-495f-9514-82e883318c90-kube-api-access-lthnm\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948022    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948037    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-registration-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948046    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-cni-multus\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948110    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-systemd\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948133    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-cni-multus\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948135    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-var-lib-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948151    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-systemd\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948175    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qjx5\" (UniqueName: \"kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-kube-api-access-8qjx5\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948168    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-var-lib-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948198    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-netns\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948176    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-web-config\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948240    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-cni-netd\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948281    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90836408-13df-4995-acec-b7bf40eeec84-host\") pod \"node-ca-2n8jc\" (UID: \"90836408-13df-4995-acec-b7bf40eeec84\") " pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948294    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-run-netns\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948308    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-kubernetes\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948337    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-host-cni-netd\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948333    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-system-cni-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948374    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948396    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-system-cni-dir\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948409    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90836408-13df-4995-acec-b7bf40eeec84-host\") pod \"node-ca-2n8jc\" (UID: \"90836408-13df-4995-acec-b7bf40eeec84\") " pod="openshift-image-registry/node-ca-2n8jc"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948413    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948438    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24476d57-ccbb-45a7-a594-5ba46b48ea41-cert\") pod \"ingress-canary-xszjb\" (UID: \"24476d57-ccbb-45a7-a594-5ba46b48ea41\") " pod="openshift-ingress-canary/ingress-canary-xszjb"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948445    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-kubernetes\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948462    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948468    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-config-out\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948497    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysconfig\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948526    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b22e16a-511b-4d70-a39c-16160ef790b1-kubelet-config\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948553    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-node-exporter-wtmp\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948579    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-systemd\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948604    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-sys\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948620    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24476d57-ccbb-45a7-a594-5ba46b48ea41-cert\") pod \"ingress-canary-xszjb\" (UID: \"24476d57-ccbb-45a7-a594-5ba46b48ea41\") " pod="openshift-ingress-canary/ingress-canary-xszjb"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948630    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysctl-d\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948649    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948657    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b22e16a-511b-4d70-a39c-16160ef790b1-kubelet-config\") pod \"global-pull-secret-syncer-lqnwg\" (UID: \"7b22e16a-511b-4d70-a39c-16160ef790b1\") " pod="kube-system/global-pull-secret-syncer-lqnwg"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948657    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948718    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-systemd\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948736    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-config-out\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948742    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-socket-dir-parent\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948784    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-kubelet\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948812    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948869    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysctl-d\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948897    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-host-var-lib-kubelet\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948899    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948905    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-etc-sysconfig\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948899    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98108a97-4bce-4291-8c25-ef6a88d542e4-multus-socket-dir-parent\") pod \"multus-pc8t6\" (UID: \"98108a97-4bce-4291-8c25-ef6a88d542e4\") " pod="openshift-multus/multus-pc8t6"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.948939    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.949019    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.949051    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-host\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.949090    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.949091    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb8251a8-f48e-4af4-9606-5cbdea4a6b96-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s5hhz\" (UID: \"cb8251a8-f48e-4af4-9606-5cbdea4a6b96\") " pod="openshift-multus/multus-additional-cni-plugins-s5hhz"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.949104    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.949140    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.949152    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8273a1a4-5dbb-4f54-8f4c-4207cc11c228-host\") pod \"tuned-6ksh4\" (UID: \"8273a1a4-5dbb-4f54-8f4c-4207cc11c228\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksh4"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.949162    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-openvswitch\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.949167    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-ovn\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.949196    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-run-ovn\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.949210    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-node-log\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.949236    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-etc-selinux\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.949216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.949258    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-config-volume\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.951623 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.949309    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98cda745-c0e3-45ce-bcb0-5e99dea4f1e4-node-log\") pod \"ovnkube-node-fqmk8\" (UID: \"98cda745-c0e3-45ce-bcb0-5e99dea4f1e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:24.951623 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.949334    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0efd5bb-5c99-4c84-a77f-d816995e6f79-etc-selinux\") pod \"aws-ebs-csi-driver-node-qhzg5\" (UID: \"f0efd5bb-5c99-4c84-a77f-d816995e6f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhzg5"
Nov 03 19:27:24.951623 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.949490    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-config-volume\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:24.958759 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.958738    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b45zq\" (UniqueName: \"kubernetes.io/projected/7d79046e-eee4-4336-a852-131f013288a7-kube-api-access-b45zq\") pod \"iptables-alerter-sp6pz\" (UID: \"7d79046e-eee4-4336-a852-131f013288a7\") " pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:27:24.959035 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.958980    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdspl\" (UniqueName: \"kubernetes.io/projected/f9b95482-e639-4d23-97a8-59941d9ecbe4-kube-api-access-bdspl\") pod \"metrics-server-7d8d7db6bc-592nt\" (UID: \"f9b95482-e639-4d23-97a8-59941d9ecbe4\") " pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:24.959155 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.959139    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql6s2\" (UniqueName: \"kubernetes.io/projected/65df60f4-e551-4351-a403-26305bece450-kube-api-access-ql6s2\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:24.959317 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.959296    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lthnm\" (UniqueName: \"kubernetes.io/projected/2f6a7918-e7d4-495f-9514-82e883318c90-kube-api-access-lthnm\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:24.959493 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.959475    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlkr2\" (UniqueName: \"kubernetes.io/projected/4ea78abe-c045-41df-b4c8-d5275dcb19e4-kube-api-access-xlkr2\") pod \"dns-default-h6nf4\" (UID: \"4ea78abe-c045-41df-b4c8-d5275dcb19e4\") " pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:24.963930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.963908    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl729\" (UniqueName: \"kubernetes.io/projected/aef23c0d-d927-4a6e-bc22-288027b498a8-kube-api-access-hl729\") pod \"network-check-target-494mb\" (UID: \"aef23c0d-d927-4a6e-bc22-288027b498a8\") " pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:27:24.964082 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.964065    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxzs\" (UniqueName: \"kubernetes.io/projected/24476d57-ccbb-45a7-a594-5ba46b48ea41-kube-api-access-twxzs\") pod \"ingress-canary-xszjb\" (UID: \"24476d57-ccbb-45a7-a594-5ba46b48ea41\") " pod="openshift-ingress-canary/ingress-canary-xszjb"
Nov 03 19:27:24.964444 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:24.964425    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qjx5\" (UniqueName: \"kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-kube-api-access-8qjx5\") pod \"alertmanager-main-0\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:25.049993 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:25.049958    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-node-exporter-wtmp\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:25.049993 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:25.049993    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-sys\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:25.050186 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:25.050016    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-root\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:25.050186 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:25.050042    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d79046e-eee4-4336-a852-131f013288a7-host-slash\") pod \"iptables-alerter-sp6pz\" (UID: \"7d79046e-eee4-4336-a852-131f013288a7\") " pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:27:25.050186 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:25.050061    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2f6a7918-e7d4-495f-9514-82e883318c90-crio-socket\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:25.050186 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:25.050101    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-sys\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:25.050186 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:25.050128    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-root\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:25.050186 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:25.050142    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2f6a7918-e7d4-495f-9514-82e883318c90-crio-socket\") pod \"insights-runtime-extractor-hkjp4\" (UID: \"2f6a7918-e7d4-495f-9514-82e883318c90\") " pod="openshift-insights/insights-runtime-extractor-hkjp4"
Nov 03 19:27:25.050186 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:25.050109    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d79046e-eee4-4336-a852-131f013288a7-host-slash\") pod \"iptables-alerter-sp6pz\" (UID: \"7d79046e-eee4-4336-a852-131f013288a7\") " pod="openshift-network-operator/iptables-alerter-sp6pz"
Nov 03 19:27:25.050186 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:25.050112    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/65df60f4-e551-4351-a403-26305bece450-node-exporter-wtmp\") pod \"node-exporter-z24qr\" (UID: \"65df60f4-e551-4351-a403-26305bece450\") " pod="openshift-monitoring/node-exporter-z24qr"
Nov 03 19:27:25.860969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:25.860934    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:27:26.382309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:26.382277    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h6nf4"
Nov 03 19:27:27.206573 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:27.206548    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:28.386233 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:28.386199    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:28.555079 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:28.555048    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:29.429571 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:29.429545    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:29.429966 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:29.429751    5237 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:27:35.910185 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:35.910099    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:36.081100 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:36.081063    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:37.662787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:37.662725    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:37.801578 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:37.801549    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:38.804370 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:38.804342    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-52ft7"
Nov 03 19:27:43.254464 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:27:43.254426    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:27:44.376761 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:44.376725    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7d8d7db6bc-592nt"
Nov 03 19:27:46.490563 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:46.490535    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqmk8"
Nov 03 19:27:48.592652 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:48.592614    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:48.735309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:48.735263    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:49.595873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:49.595822    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:27:49.596291 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:49.596198    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="kube-rbac-proxy-metric" containerID="cri-o://27b8a439469acba6bebe5dade52a297a012b9a930e9bf6d686fca8a2051184e2" gracePeriod=120
Nov 03 19:27:49.596291 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:49.596243    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="kube-rbac-proxy-web" containerID="cri-o://7d6c8a7691de319d2ae6933044d9fdf35e430df69babdd9abb404944d4e1ae79" gracePeriod=120
Nov 03 19:27:49.596291 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:49.596253    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="config-reloader" containerID="cri-o://aa1f57f1d141703876722bc27cf46fc511beee32c35e05d749f518dd9a0cc484" gracePeriod=120
Nov 03 19:27:49.596603 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:49.596271    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="prom-label-proxy" containerID="cri-o://e213c3ca058418f708f618a1c9b13d4ed8b2eee740394dc5e556ccfa08ec0977" gracePeriod=120
Nov 03 19:27:49.596603 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:49.596185    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="alertmanager" containerID="cri-o://91f015b7d832ff6c64a800d3cb643ea563f3c8be16d6463a28046a392d128f0d" gracePeriod=120
Nov 03 19:27:49.596603 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:49.596351    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="kube-rbac-proxy" containerID="cri-o://eb2ae76c62df0d09b240aea79a3a7c77698e25c9406fbbfae9542b9810b9e285" gracePeriod=120
Nov 03 19:27:49.983547 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:49.983469    5237 generic.go:356] "Generic (PLEG): container finished" podID="0827774e-9562-487f-9c6e-0934ef8707ea" containerID="e213c3ca058418f708f618a1c9b13d4ed8b2eee740394dc5e556ccfa08ec0977" exitCode=0
Nov 03 19:27:49.983547 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:49.983494    5237 generic.go:356] "Generic (PLEG): container finished" podID="0827774e-9562-487f-9c6e-0934ef8707ea" containerID="27b8a439469acba6bebe5dade52a297a012b9a930e9bf6d686fca8a2051184e2" exitCode=0
Nov 03 19:27:49.983547 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:49.983501    5237 generic.go:356] "Generic (PLEG): container finished" podID="0827774e-9562-487f-9c6e-0934ef8707ea" containerID="eb2ae76c62df0d09b240aea79a3a7c77698e25c9406fbbfae9542b9810b9e285" exitCode=0
Nov 03 19:27:49.983547 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:49.983507    5237 generic.go:356] "Generic (PLEG): container finished" podID="0827774e-9562-487f-9c6e-0934ef8707ea" containerID="aa1f57f1d141703876722bc27cf46fc511beee32c35e05d749f518dd9a0cc484" exitCode=0
Nov 03 19:27:49.983547 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:49.983513    5237 generic.go:356] "Generic (PLEG): container finished" podID="0827774e-9562-487f-9c6e-0934ef8707ea" containerID="91f015b7d832ff6c64a800d3cb643ea563f3c8be16d6463a28046a392d128f0d" exitCode=0
Nov 03 19:27:49.983805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:49.983540    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0827774e-9562-487f-9c6e-0934ef8707ea","Type":"ContainerDied","Data":"e213c3ca058418f708f618a1c9b13d4ed8b2eee740394dc5e556ccfa08ec0977"}
Nov 03 19:27:49.983805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:49.983613    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0827774e-9562-487f-9c6e-0934ef8707ea","Type":"ContainerDied","Data":"27b8a439469acba6bebe5dade52a297a012b9a930e9bf6d686fca8a2051184e2"}
Nov 03 19:27:49.983805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:49.983640    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0827774e-9562-487f-9c6e-0934ef8707ea","Type":"ContainerDied","Data":"eb2ae76c62df0d09b240aea79a3a7c77698e25c9406fbbfae9542b9810b9e285"}
Nov 03 19:27:49.983805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:49.983673    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0827774e-9562-487f-9c6e-0934ef8707ea","Type":"ContainerDied","Data":"aa1f57f1d141703876722bc27cf46fc511beee32c35e05d749f518dd9a0cc484"}
Nov 03 19:27:49.983805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:49.983696    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0827774e-9562-487f-9c6e-0934ef8707ea","Type":"ContainerDied","Data":"91f015b7d832ff6c64a800d3cb643ea563f3c8be16d6463a28046a392d128f0d"}
Nov 03 19:27:50.830427 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.830406    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:50.918382 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.918342    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:50.922214 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.922193    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-config-volume\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-config-volume\") pod \"0827774e-9562-487f-9c6e-0934ef8707ea\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") "
Nov 03 19:27:50.922275 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.922229    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-kube-api-access-8qjx5\" (UniqueName: \"kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-kube-api-access-8qjx5\") pod \"0827774e-9562-487f-9c6e-0934ef8707ea\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") "
Nov 03 19:27:50.922275 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.922255    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-cluster-tls-config\") pod \"0827774e-9562-487f-9c6e-0934ef8707ea\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") "
Nov 03 19:27:50.922360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.922279    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-main-db\") pod \"0827774e-9562-487f-9c6e-0934ef8707ea\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") "
Nov 03 19:27:50.922360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.922300    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-main-tls\") pod \"0827774e-9562-487f-9c6e-0934ef8707ea\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") "
Nov 03 19:27:50.922360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.922324    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-web\") pod \"0827774e-9562-487f-9c6e-0934ef8707ea\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") "
Nov 03 19:27:50.922360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.922347    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-trusted-ca-bundle\") pod \"0827774e-9562-487f-9c6e-0934ef8707ea\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") "
Nov 03 19:27:50.922567 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.922386    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-metrics-client-ca\") pod \"0827774e-9562-487f-9c6e-0934ef8707ea\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") "
Nov 03 19:27:50.922567 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.922428    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy\") pod \"0827774e-9562-487f-9c6e-0934ef8707ea\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") "
Nov 03 19:27:50.922567 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.922468    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-config-out\") pod \"0827774e-9562-487f-9c6e-0934ef8707ea\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") "
Nov 03 19:27:50.922567 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.922496    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-web-config\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-web-config\") pod \"0827774e-9562-487f-9c6e-0934ef8707ea\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") "
Nov 03 19:27:50.922567 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.922536    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0827774e-9562-487f-9c6e-0934ef8707ea\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") "
Nov 03 19:27:50.922824 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.922585    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-tls-assets\" (UniqueName: \"kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-tls-assets\") pod \"0827774e-9562-487f-9c6e-0934ef8707ea\" (UID: \"0827774e-9562-487f-9c6e-0934ef8707ea\") "
Nov 03 19:27:50.922824 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.922618    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-main-db" pod "0827774e-9562-487f-9c6e-0934ef8707ea" (UID: "0827774e-9562-487f-9c6e-0934ef8707ea"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:27:50.922824 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.922804    5237 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-main-db\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:50.924334 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.924300    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-metrics-client-ca" pod "0827774e-9562-487f-9c6e-0934ef8707ea" (UID: "0827774e-9562-487f-9c6e-0934ef8707ea"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:27:50.924518 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.924489    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-trusted-ca-bundle" pod "0827774e-9562-487f-9c6e-0934ef8707ea" (UID: "0827774e-9562-487f-9c6e-0934ef8707ea"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:27:50.926981 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.926925    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-main-tls" pod "0827774e-9562-487f-9c6e-0934ef8707ea" (UID: "0827774e-9562-487f-9c6e-0934ef8707ea"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:27:50.927077 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.927050    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-tls-assets" pod "0827774e-9562-487f-9c6e-0934ef8707ea" (UID: "0827774e-9562-487f-9c6e-0934ef8707ea"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:27:50.927152 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.927054    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-config-out" pod "0827774e-9562-487f-9c6e-0934ef8707ea" (UID: "0827774e-9562-487f-9c6e-0934ef8707ea"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:27:50.927209 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.927144    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy" pod "0827774e-9562-487f-9c6e-0934ef8707ea" (UID: "0827774e-9562-487f-9c6e-0934ef8707ea"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:27:50.927209 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.927153    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-kube-api-access-8qjx5" pod "0827774e-9562-487f-9c6e-0934ef8707ea" (UID: "0827774e-9562-487f-9c6e-0934ef8707ea"). InnerVolumeSpecName "kube-api-access-8qjx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:27:50.927568 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.927545    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-metric" pod "0827774e-9562-487f-9c6e-0934ef8707ea" (UID: "0827774e-9562-487f-9c6e-0934ef8707ea"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:27:50.927662 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.927639    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-config-volume" pod "0827774e-9562-487f-9c6e-0934ef8707ea" (UID: "0827774e-9562-487f-9c6e-0934ef8707ea"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:27:50.927749 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.927730    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-web" pod "0827774e-9562-487f-9c6e-0934ef8707ea" (UID: "0827774e-9562-487f-9c6e-0934ef8707ea"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:27:50.930509 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.930485    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-cluster-tls-config" pod "0827774e-9562-487f-9c6e-0934ef8707ea" (UID: "0827774e-9562-487f-9c6e-0934ef8707ea"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:27:50.936637 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.936606    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-web-config" pod "0827774e-9562-487f-9c6e-0934ef8707ea" (UID: "0827774e-9562-487f-9c6e-0934ef8707ea"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:27:50.992775 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.992738    5237 generic.go:356] "Generic (PLEG): container finished" podID="0827774e-9562-487f-9c6e-0934ef8707ea" containerID="7d6c8a7691de319d2ae6933044d9fdf35e430df69babdd9abb404944d4e1ae79" exitCode=0
Nov 03 19:27:50.992987 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.992808    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0827774e-9562-487f-9c6e-0934ef8707ea","Type":"ContainerDied","Data":"7d6c8a7691de319d2ae6933044d9fdf35e430df69babdd9abb404944d4e1ae79"}
Nov 03 19:27:50.992987 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.992857    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:50.992987 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.992907    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0827774e-9562-487f-9c6e-0934ef8707ea","Type":"ContainerDied","Data":"90383a53ed9782bf968aff14e6c06a34b754360b77ab2524c4a17df8fc66c857"}
Nov 03 19:27:50.992987 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:50.992943    5237 scope.go:122] "RemoveContainer" containerID="e213c3ca058418f708f618a1c9b13d4ed8b2eee740394dc5e556ccfa08ec0977"
Nov 03 19:27:51.001239 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.001216    5237 scope.go:122] "RemoveContainer" containerID="27b8a439469acba6bebe5dade52a297a012b9a930e9bf6d686fca8a2051184e2"
Nov 03 19:27:51.008109 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.008087    5237 scope.go:122] "RemoveContainer" containerID="eb2ae76c62df0d09b240aea79a3a7c77698e25c9406fbbfae9542b9810b9e285"
Nov 03 19:27:51.014434 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.014413    5237 scope.go:122] "RemoveContainer" containerID="7d6c8a7691de319d2ae6933044d9fdf35e430df69babdd9abb404944d4e1ae79"
Nov 03 19:27:51.020575 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.020558    5237 scope.go:122] "RemoveContainer" containerID="aa1f57f1d141703876722bc27cf46fc511beee32c35e05d749f518dd9a0cc484"
Nov 03 19:27:51.023325 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.023304    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:27:51.023537 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.023518    5237 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-tls-assets\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:51.023575 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.023542    5237 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-config-volume\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:51.023575 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.023554    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qjx5\" (UniqueName: \"kubernetes.io/projected/0827774e-9562-487f-9c6e-0934ef8707ea-kube-api-access-8qjx5\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:51.023575 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.023565    5237 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-cluster-tls-config\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:51.023670 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.023575    5237 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-main-tls\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:51.023670 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.023585    5237 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:51.023670 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.023595    5237 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:51.023670 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.023609    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0827774e-9562-487f-9c6e-0934ef8707ea-metrics-client-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:51.023670 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.023619    5237 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:51.023670 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.023627    5237 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0827774e-9562-487f-9c6e-0934ef8707ea-config-out\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:51.023670 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.023655    5237 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-web-config\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:51.023670 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.023664    5237 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0827774e-9562-487f-9c6e-0934ef8707ea-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:51.027488 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.027466    5237 scope.go:122] "RemoveContainer" containerID="91f015b7d832ff6c64a800d3cb643ea563f3c8be16d6463a28046a392d128f0d"
Nov 03 19:27:51.031261 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.031239    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:27:51.034593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.034570    5237 scope.go:122] "RemoveContainer" containerID="dcc926ed375a62f53970a52e5d65852dbdcb9cda977e99db05668588f549c204"
Nov 03 19:27:51.040974 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.040958    5237 scope.go:122] "RemoveContainer" containerID="e213c3ca058418f708f618a1c9b13d4ed8b2eee740394dc5e556ccfa08ec0977"
Nov 03 19:27:51.041247 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:27:51.041225    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e213c3ca058418f708f618a1c9b13d4ed8b2eee740394dc5e556ccfa08ec0977\": container with ID starting with e213c3ca058418f708f618a1c9b13d4ed8b2eee740394dc5e556ccfa08ec0977 not found: ID does not exist" containerID="e213c3ca058418f708f618a1c9b13d4ed8b2eee740394dc5e556ccfa08ec0977"
Nov 03 19:27:51.041295 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.041260    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e213c3ca058418f708f618a1c9b13d4ed8b2eee740394dc5e556ccfa08ec0977"} err="failed to get container status \"e213c3ca058418f708f618a1c9b13d4ed8b2eee740394dc5e556ccfa08ec0977\": rpc error: code = NotFound desc = could not find container \"e213c3ca058418f708f618a1c9b13d4ed8b2eee740394dc5e556ccfa08ec0977\": container with ID starting with e213c3ca058418f708f618a1c9b13d4ed8b2eee740394dc5e556ccfa08ec0977 not found: ID does not exist"
Nov 03 19:27:51.041336 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.041297    5237 scope.go:122] "RemoveContainer" containerID="27b8a439469acba6bebe5dade52a297a012b9a930e9bf6d686fca8a2051184e2"
Nov 03 19:27:51.041528 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:27:51.041513    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b8a439469acba6bebe5dade52a297a012b9a930e9bf6d686fca8a2051184e2\": container with ID starting with 27b8a439469acba6bebe5dade52a297a012b9a930e9bf6d686fca8a2051184e2 not found: ID does not exist" containerID="27b8a439469acba6bebe5dade52a297a012b9a930e9bf6d686fca8a2051184e2"
Nov 03 19:27:51.041579 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.041532    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b8a439469acba6bebe5dade52a297a012b9a930e9bf6d686fca8a2051184e2"} err="failed to get container status \"27b8a439469acba6bebe5dade52a297a012b9a930e9bf6d686fca8a2051184e2\": rpc error: code = NotFound desc = could not find container \"27b8a439469acba6bebe5dade52a297a012b9a930e9bf6d686fca8a2051184e2\": container with ID starting with 27b8a439469acba6bebe5dade52a297a012b9a930e9bf6d686fca8a2051184e2 not found: ID does not exist"
Nov 03 19:27:51.041579 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.041543    5237 scope.go:122] "RemoveContainer" containerID="eb2ae76c62df0d09b240aea79a3a7c77698e25c9406fbbfae9542b9810b9e285"
Nov 03 19:27:51.041765 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:27:51.041747    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb2ae76c62df0d09b240aea79a3a7c77698e25c9406fbbfae9542b9810b9e285\": container with ID starting with eb2ae76c62df0d09b240aea79a3a7c77698e25c9406fbbfae9542b9810b9e285 not found: ID does not exist" containerID="eb2ae76c62df0d09b240aea79a3a7c77698e25c9406fbbfae9542b9810b9e285"
Nov 03 19:27:51.041822 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.041775    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb2ae76c62df0d09b240aea79a3a7c77698e25c9406fbbfae9542b9810b9e285"} err="failed to get container status \"eb2ae76c62df0d09b240aea79a3a7c77698e25c9406fbbfae9542b9810b9e285\": rpc error: code = NotFound desc = could not find container \"eb2ae76c62df0d09b240aea79a3a7c77698e25c9406fbbfae9542b9810b9e285\": container with ID starting with eb2ae76c62df0d09b240aea79a3a7c77698e25c9406fbbfae9542b9810b9e285 not found: ID does not exist"
Nov 03 19:27:51.041822 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.041802    5237 scope.go:122] "RemoveContainer" containerID="7d6c8a7691de319d2ae6933044d9fdf35e430df69babdd9abb404944d4e1ae79"
Nov 03 19:27:51.042083 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:27:51.042067    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d6c8a7691de319d2ae6933044d9fdf35e430df69babdd9abb404944d4e1ae79\": container with ID starting with 7d6c8a7691de319d2ae6933044d9fdf35e430df69babdd9abb404944d4e1ae79 not found: ID does not exist" containerID="7d6c8a7691de319d2ae6933044d9fdf35e430df69babdd9abb404944d4e1ae79"
Nov 03 19:27:51.042137 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.042090    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d6c8a7691de319d2ae6933044d9fdf35e430df69babdd9abb404944d4e1ae79"} err="failed to get container status \"7d6c8a7691de319d2ae6933044d9fdf35e430df69babdd9abb404944d4e1ae79\": rpc error: code = NotFound desc = could not find container \"7d6c8a7691de319d2ae6933044d9fdf35e430df69babdd9abb404944d4e1ae79\": container with ID starting with 7d6c8a7691de319d2ae6933044d9fdf35e430df69babdd9abb404944d4e1ae79 not found: ID does not exist"
Nov 03 19:27:51.042137 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.042108    5237 scope.go:122] "RemoveContainer" containerID="aa1f57f1d141703876722bc27cf46fc511beee32c35e05d749f518dd9a0cc484"
Nov 03 19:27:51.042340 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:27:51.042324    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1f57f1d141703876722bc27cf46fc511beee32c35e05d749f518dd9a0cc484\": container with ID starting with aa1f57f1d141703876722bc27cf46fc511beee32c35e05d749f518dd9a0cc484 not found: ID does not exist" containerID="aa1f57f1d141703876722bc27cf46fc511beee32c35e05d749f518dd9a0cc484"
Nov 03 19:27:51.042380 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.042344    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1f57f1d141703876722bc27cf46fc511beee32c35e05d749f518dd9a0cc484"} err="failed to get container status \"aa1f57f1d141703876722bc27cf46fc511beee32c35e05d749f518dd9a0cc484\": rpc error: code = NotFound desc = could not find container \"aa1f57f1d141703876722bc27cf46fc511beee32c35e05d749f518dd9a0cc484\": container with ID starting with aa1f57f1d141703876722bc27cf46fc511beee32c35e05d749f518dd9a0cc484 not found: ID does not exist"
Nov 03 19:27:51.042380 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.042358    5237 scope.go:122] "RemoveContainer" containerID="91f015b7d832ff6c64a800d3cb643ea563f3c8be16d6463a28046a392d128f0d"
Nov 03 19:27:51.042549 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:27:51.042535    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f015b7d832ff6c64a800d3cb643ea563f3c8be16d6463a28046a392d128f0d\": container with ID starting with 91f015b7d832ff6c64a800d3cb643ea563f3c8be16d6463a28046a392d128f0d not found: ID does not exist" containerID="91f015b7d832ff6c64a800d3cb643ea563f3c8be16d6463a28046a392d128f0d"
Nov 03 19:27:51.042588 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.042550    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f015b7d832ff6c64a800d3cb643ea563f3c8be16d6463a28046a392d128f0d"} err="failed to get container status \"91f015b7d832ff6c64a800d3cb643ea563f3c8be16d6463a28046a392d128f0d\": rpc error: code = NotFound desc = could not find container \"91f015b7d832ff6c64a800d3cb643ea563f3c8be16d6463a28046a392d128f0d\": container with ID starting with 91f015b7d832ff6c64a800d3cb643ea563f3c8be16d6463a28046a392d128f0d not found: ID does not exist"
Nov 03 19:27:51.042588 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.042561    5237 scope.go:122] "RemoveContainer" containerID="dcc926ed375a62f53970a52e5d65852dbdcb9cda977e99db05668588f549c204"
Nov 03 19:27:51.042762 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:27:51.042744    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc926ed375a62f53970a52e5d65852dbdcb9cda977e99db05668588f549c204\": container with ID starting with dcc926ed375a62f53970a52e5d65852dbdcb9cda977e99db05668588f549c204 not found: ID does not exist" containerID="dcc926ed375a62f53970a52e5d65852dbdcb9cda977e99db05668588f549c204"
Nov 03 19:27:51.042798 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.042767    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc926ed375a62f53970a52e5d65852dbdcb9cda977e99db05668588f549c204"} err="failed to get container status \"dcc926ed375a62f53970a52e5d65852dbdcb9cda977e99db05668588f549c204\": rpc error: code = NotFound desc = could not find container \"dcc926ed375a62f53970a52e5d65852dbdcb9cda977e99db05668588f549c204\": container with ID starting with dcc926ed375a62f53970a52e5d65852dbdcb9cda977e99db05668588f549c204 not found: ID does not exist"
Nov 03 19:27:51.064212 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064190    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:27:51.064594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064404    5237 cpu_manager.go:414] "containerMap: removing container" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="kube-rbac-proxy-metric"
Nov 03 19:27:51.064594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064425    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="kube-rbac-proxy-metric"
Nov 03 19:27:51.064594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064437    5237 cpu_manager.go:414] "containerMap: removing container" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="kube-rbac-proxy"
Nov 03 19:27:51.064594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064443    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="kube-rbac-proxy"
Nov 03 19:27:51.064594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064452    5237 cpu_manager.go:414] "containerMap: removing container" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="kube-rbac-proxy-web"
Nov 03 19:27:51.064594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064460    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="kube-rbac-proxy-web"
Nov 03 19:27:51.064594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064470    5237 cpu_manager.go:414] "containerMap: removing container" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="config-reloader"
Nov 03 19:27:51.064594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064477    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="config-reloader"
Nov 03 19:27:51.064594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064499    5237 cpu_manager.go:414] "containerMap: removing container" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="init-config-reloader"
Nov 03 19:27:51.064594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064506    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="init-config-reloader"
Nov 03 19:27:51.064594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064517    5237 cpu_manager.go:414] "containerMap: removing container" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="prom-label-proxy"
Nov 03 19:27:51.064594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064526    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="prom-label-proxy"
Nov 03 19:27:51.064594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064543    5237 cpu_manager.go:414] "containerMap: removing container" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="alertmanager"
Nov 03 19:27:51.064594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064549    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="alertmanager"
Nov 03 19:27:51.065103 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064607    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="kube-rbac-proxy-metric"
Nov 03 19:27:51.065103 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064617    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="kube-rbac-proxy"
Nov 03 19:27:51.065103 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064626    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="kube-rbac-proxy-web"
Nov 03 19:27:51.065103 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064634    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="config-reloader"
Nov 03 19:27:51.065103 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064645    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="init-config-reloader"
Nov 03 19:27:51.065103 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064652    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="prom-label-proxy"
Nov 03 19:27:51.065103 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.064660    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" containerName="alertmanager"
Nov 03 19:27:51.067623 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.067609    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.074281 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.074209    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 19:27:51.074281 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.074270    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 19:27:51.074281 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.074278    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 19:27:51.074445 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.074284    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 19:27:51.074445 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.074281    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 19:27:51.074445 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.074328    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 19:27:51.074627 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.074552    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kw62n\""
Nov 03 19:27:51.075229 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.075213    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 19:27:51.076616 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.076603    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 19:27:51.078542 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.078520    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:51.081075 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.081057    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 19:27:51.092337 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.092316    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:27:51.225401 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.225311    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-web-config\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.225401 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.225360    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.225584 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.225421    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.225584 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.225467    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.225584 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.225487    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b945ca1-7ff9-4062-aede-430a2061c9db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.225584 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.225526    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgrqk\" (UniqueName: \"kubernetes.io/projected/6b945ca1-7ff9-4062-aede-430a2061c9db-kube-api-access-zgrqk\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.225728 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.225609    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.225728 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.225651    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.225728 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.225670    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b945ca1-7ff9-4062-aede-430a2061c9db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.225876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.225736    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b945ca1-7ff9-4062-aede-430a2061c9db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.225876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.225757    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.225876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.225779    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b945ca1-7ff9-4062-aede-430a2061c9db-config-out\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.225876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.225860    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b945ca1-7ff9-4062-aede-430a2061c9db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.326590 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.326546    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b945ca1-7ff9-4062-aede-430a2061c9db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.326590 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.326594    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-web-config\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.326831 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.326629    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.326831 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.326658    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.326831 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.326686    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.326831 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.326708    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b945ca1-7ff9-4062-aede-430a2061c9db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.326831 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.326734    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgrqk\" (UniqueName: \"kubernetes.io/projected/6b945ca1-7ff9-4062-aede-430a2061c9db-kube-api-access-zgrqk\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.326831 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.326765    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.326831 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.326801    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.326831 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.326826    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b945ca1-7ff9-4062-aede-430a2061c9db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.327251 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.326884    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b945ca1-7ff9-4062-aede-430a2061c9db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.327251 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.326914    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.327251 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.326948    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b945ca1-7ff9-4062-aede-430a2061c9db-config-out\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.327251 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.327063    5237 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 19:27:51.327251 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.327116    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b945ca1-7ff9-4062-aede-430a2061c9db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.327496 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.327410    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b945ca1-7ff9-4062-aede-430a2061c9db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.327829 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.327803    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b945ca1-7ff9-4062-aede-430a2061c9db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.329783 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.329752    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b945ca1-7ff9-4062-aede-430a2061c9db-config-out\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.329902 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.329879    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.330154 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.330135    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.330154 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.330150    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.330281 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.330158    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.330281 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.330264    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.330344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.330292    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-web-config\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.330344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.330268    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b945ca1-7ff9-4062-aede-430a2061c9db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.331546 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.331529    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b945ca1-7ff9-4062-aede-430a2061c9db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.339336 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.339314    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgrqk\" (UniqueName: \"kubernetes.io/projected/6b945ca1-7ff9-4062-aede-430a2061c9db-kube-api-access-zgrqk\") pod \"alertmanager-main-0\" (UID: \"6b945ca1-7ff9-4062-aede-430a2061c9db\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.383369 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.383329    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:27:51.666858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.666809    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:27:51.670060 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:51.670029    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b945ca1_7ff9_4062_aede_430a2061c9db.slice/crio-7862247074d99b2c4e574d123b2ac0618704a462a56f06f0e11e132d282a7858 WatchSource:0}: Error finding container 7862247074d99b2c4e574d123b2ac0618704a462a56f06f0e11e132d282a7858: Status 404 returned error can't find the container with id 7862247074d99b2c4e574d123b2ac0618704a462a56f06f0e11e132d282a7858
Nov 03 19:27:51.856486 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.856445    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0827774e-9562-487f-9c6e-0934ef8707ea" path="/var/lib/kubelet/pods/0827774e-9562-487f-9c6e-0934ef8707ea/volumes"
Nov 03 19:27:51.998382 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.998297    5237 generic.go:356] "Generic (PLEG): container finished" podID="6b945ca1-7ff9-4062-aede-430a2061c9db" containerID="4c223ea9453ccc7acd44c12136845f92f46e882c067defe41657a5769422f01d" exitCode=0
Nov 03 19:27:51.998382 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.998354    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b945ca1-7ff9-4062-aede-430a2061c9db","Type":"ContainerDied","Data":"4c223ea9453ccc7acd44c12136845f92f46e882c067defe41657a5769422f01d"}
Nov 03 19:27:51.998570 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:51.998403    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b945ca1-7ff9-4062-aede-430a2061c9db","Type":"ContainerStarted","Data":"7862247074d99b2c4e574d123b2ac0618704a462a56f06f0e11e132d282a7858"}
Nov 03 19:27:53.004530 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.004492    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b945ca1-7ff9-4062-aede-430a2061c9db","Type":"ContainerStarted","Data":"7dc5c94dcc322233bd149d4edbed7f93b83b9ae6ce29a4db0cd2ca1e9c369c6e"}
Nov 03 19:27:53.004910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.004611    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b945ca1-7ff9-4062-aede-430a2061c9db","Type":"ContainerStarted","Data":"84cf9c70c6f129b59912f5eef01f26da0b6288c50bb45f5fcdd9fc156dccf94b"}
Nov 03 19:27:53.004910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.004656    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b945ca1-7ff9-4062-aede-430a2061c9db","Type":"ContainerStarted","Data":"9259d9526d7c1f616c70185ba26942347636d957a4123911fc7a5b81b20f03f8"}
Nov 03 19:27:53.004910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.004694    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b945ca1-7ff9-4062-aede-430a2061c9db","Type":"ContainerStarted","Data":"dfb6c2d507d32f772ff80fe543053a54721934753c4eb6cbb19f37888b6603d0"}
Nov 03 19:27:53.004910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.004729    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b945ca1-7ff9-4062-aede-430a2061c9db","Type":"ContainerStarted","Data":"470615450f22a5208af25160f2655967b8eaeb7510a4d4102693295e327b22af"}
Nov 03 19:27:53.004910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.004764    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b945ca1-7ff9-4062-aede-430a2061c9db","Type":"ContainerStarted","Data":"4e54fcff3c43418cd8039e8cb04ff2ec88591f68e31a3b141942fb3f205c4f94"}
Nov 03 19:27:53.039671 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.039617    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.024550497 podStartE2EDuration="2.039603997s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.0535ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:53.038252041 +0000 UTC m=+29.292559268" watchObservedRunningTime="2026-11-03 19:27:53.039603997 +0000 UTC m=+29.293911221"
Nov 03 19:27:53.704503 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.704469    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"]
Nov 03 19:27:53.706793 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.706772    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.712784 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.712758    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 19:27:53.712784 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.712756    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 19:27:53.713009 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.712798    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 19:27:53.713009 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.712806    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 19:27:53.713009 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.712804    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bl2kw\""
Nov 03 19:27:53.722883 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.722861    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"]
Nov 03 19:27:53.793259 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.793216    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 19:27:53.795708 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.795689    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 19:27:53.850417 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.850380    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5ed038d-cacd-4ec2-b755-d8e714aef422-serving-certs-ca-bundle\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.850417 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.850423    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5ed038d-cacd-4ec2-b755-d8e714aef422-metrics-client-ca\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.850685 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.850449    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgnbx\" (UniqueName: \"kubernetes.io/projected/e5ed038d-cacd-4ec2-b755-d8e714aef422-kube-api-access-zgnbx\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.850685 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.850470    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5ed038d-cacd-4ec2-b755-d8e714aef422-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.850685 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.850565    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5ed038d-cacd-4ec2-b755-d8e714aef422-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.850685 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.850609    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e5ed038d-cacd-4ec2-b755-d8e714aef422-telemeter-client-tls\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.850685 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.850639    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e5ed038d-cacd-4ec2-b755-d8e714aef422-federate-client-tls\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.850685 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.850658    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e5ed038d-cacd-4ec2-b755-d8e714aef422-secret-telemeter-client\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.951445 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.951405    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5ed038d-cacd-4ec2-b755-d8e714aef422-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.951601 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.951449    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5ed038d-cacd-4ec2-b755-d8e714aef422-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.951601 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.951484    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e5ed038d-cacd-4ec2-b755-d8e714aef422-telemeter-client-tls\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.951601 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.951524    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e5ed038d-cacd-4ec2-b755-d8e714aef422-federate-client-tls\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.951601 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.951556    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e5ed038d-cacd-4ec2-b755-d8e714aef422-secret-telemeter-client\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.951805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.951614    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5ed038d-cacd-4ec2-b755-d8e714aef422-serving-certs-ca-bundle\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.951805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.951652    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5ed038d-cacd-4ec2-b755-d8e714aef422-metrics-client-ca\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.951805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.951674    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgnbx\" (UniqueName: \"kubernetes.io/projected/e5ed038d-cacd-4ec2-b755-d8e714aef422-kube-api-access-zgnbx\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.952515 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.952480    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5ed038d-cacd-4ec2-b755-d8e714aef422-metrics-client-ca\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.952653 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.952508    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5ed038d-cacd-4ec2-b755-d8e714aef422-serving-certs-ca-bundle\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.952738 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.952711    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5ed038d-cacd-4ec2-b755-d8e714aef422-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.954400 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.954373    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e5ed038d-cacd-4ec2-b755-d8e714aef422-federate-client-tls\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.954623 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.954578    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e5ed038d-cacd-4ec2-b755-d8e714aef422-telemeter-client-tls\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.954623 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.954587    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e5ed038d-cacd-4ec2-b755-d8e714aef422-secret-telemeter-client\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.954732 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.954677    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5ed038d-cacd-4ec2-b755-d8e714aef422-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:53.964610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:53.964582    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgnbx\" (UniqueName: \"kubernetes.io/projected/e5ed038d-cacd-4ec2-b755-d8e714aef422-kube-api-access-zgnbx\") pod \"telemeter-client-7c5449fd55-vnf6b\" (UID: \"e5ed038d-cacd-4ec2-b755-d8e714aef422\") " pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:54.019671 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:54.019637    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"
Nov 03 19:27:54.304119 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:54.304059    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c5449fd55-vnf6b"]
Nov 03 19:27:54.307614 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:27:54.307568    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ed038d_cacd_4ec2_b755_d8e714aef422.slice/crio-c02c9d7356f9f558b7da1e96c099526300ad50af494a4abacf27c4b4847305af WatchSource:0}: Error finding container c02c9d7356f9f558b7da1e96c099526300ad50af494a4abacf27c4b4847305af: Status 404 returned error can't find the container with id c02c9d7356f9f558b7da1e96c099526300ad50af494a4abacf27c4b4847305af
Nov 03 19:27:54.309704 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:54.309685    5237 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:27:55.013529 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:55.013488    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b" event={"ID":"e5ed038d-cacd-4ec2-b755-d8e714aef422","Type":"ContainerStarted","Data":"c02c9d7356f9f558b7da1e96c099526300ad50af494a4abacf27c4b4847305af"}
Nov 03 19:27:55.910295 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:55.910268    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-494mb"
Nov 03 19:27:56.019351 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:56.019320    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b" event={"ID":"e5ed038d-cacd-4ec2-b755-d8e714aef422","Type":"ContainerStarted","Data":"5985e319f54ccfc00d275e16cf2125aeb50b6506f08f4bbbf590f32b4ca995de"}
Nov 03 19:27:56.019502 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:56.019372    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b" event={"ID":"e5ed038d-cacd-4ec2-b755-d8e714aef422","Type":"ContainerStarted","Data":"c4516cb077b216eae355f981ae568de20322d1af06f0189884bd3c2e444a23fa"}
Nov 03 19:27:56.019502 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:56.019395    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b" event={"ID":"e5ed038d-cacd-4ec2-b755-d8e714aef422","Type":"ContainerStarted","Data":"614a80d018f4f44ae9dbc1f913c55effbd2dfe3abfbfb6834ff5aacc0e7cc9dd"}
Nov 03 19:27:56.046774 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:56.046724    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7c5449fd55-vnf6b" podStartSLOduration=1.5337239440000001 podStartE2EDuration="3.046710202s" totalImagesPullingTime="1.512986258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:56.046609056 +0000 UTC m=+32.300916281" watchObservedRunningTime="2026-11-03 19:27:56.046710202 +0000 UTC m=+32.301017423"
Nov 03 19:27:56.294370 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:56.294339    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:56.437967 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:56.437925    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:57.151879 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:57.151824    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:57.284057 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:27:57.284019    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:13.254711 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:28:13.254671    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:28:28.445055 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:28:28.445020    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:43.255095 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:28:43.255058    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:28:44.272174 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:28:44.272137    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:53.891497 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:28:53.891461    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:54.017996 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:28:54.017962    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:06.491113 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:29:06.491014    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:06.624701 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:29:06.624657    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:06.854364 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:29:06.854329    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:06.988809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:29:06.988777    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:13.254235 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:29:13.254200    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:29:17.451301 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:29:17.451263    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:20.885784 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:29:20.885747    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:21.012805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:29:21.012773    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:28.483502 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:29:28.483457    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:30.635297 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:29:30.635255    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:30.760470 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:29:30.760439    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:38.441328 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:29:38.441285    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:43.253877 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:29:43.253820    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:30:04.466033 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:30:04.466001    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:13.255152 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:30:13.255104    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:30:28.544423 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:30:28.544381    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:43.254289 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:30:43.254206    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:30:54.178763 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:30:54.178725    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:13.255222 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:31:13.255181    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:31:28.551058 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:31:28.551026    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:36.235003 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:31:36.234967    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:43.254387 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:31:43.254341    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:32:13.254324 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:32:13.254286    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:32:28.649792 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:28.649751    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:33.560149 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:33.560113    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:33.753752 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:33.753695    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:34.468075 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.468030    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk"]
Nov 03 19:32:34.471352 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.471334    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk"
Nov 03 19:32:34.476748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.476720    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 19:32:34.476859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.476822    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 19:32:34.476909 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.476850    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-9xb7x\""
Nov 03 19:32:34.495501 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.495480    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk"]
Nov 03 19:32:34.530931 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.530897    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fzkj\" (UniqueName: \"kubernetes.io/projected/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-kube-api-access-6fzkj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk\" (UID: \"3a1c567d-45e8-4bcb-a6a7-4f11217f2299\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk"
Nov 03 19:32:34.531076 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.530953    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk\" (UID: \"3a1c567d-45e8-4bcb-a6a7-4f11217f2299\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk"
Nov 03 19:32:34.531116 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.531069    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk\" (UID: \"3a1c567d-45e8-4bcb-a6a7-4f11217f2299\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk"
Nov 03 19:32:34.566646 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.566616    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd"]
Nov 03 19:32:34.569988 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.569968    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd"
Nov 03 19:32:34.576617 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.576598    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-nf9dg\""
Nov 03 19:32:34.585169 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.585148    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd"]
Nov 03 19:32:34.631405 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.631367    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk\" (UID: \"3a1c567d-45e8-4bcb-a6a7-4f11217f2299\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk"
Nov 03 19:32:34.631587 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.631418    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fzkj\" (UniqueName: \"kubernetes.io/projected/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-kube-api-access-6fzkj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk\" (UID: \"3a1c567d-45e8-4bcb-a6a7-4f11217f2299\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk"
Nov 03 19:32:34.631587 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.631458    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3d62efc-26ac-49a6-bedf-3babdd66abe5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd\" (UID: \"c3d62efc-26ac-49a6-bedf-3babdd66abe5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd"
Nov 03 19:32:34.631587 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.631480    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74c97\" (UniqueName: \"kubernetes.io/projected/c3d62efc-26ac-49a6-bedf-3babdd66abe5-kube-api-access-74c97\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd\" (UID: \"c3d62efc-26ac-49a6-bedf-3babdd66abe5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd"
Nov 03 19:32:34.631587 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.631516    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk\" (UID: \"3a1c567d-45e8-4bcb-a6a7-4f11217f2299\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk"
Nov 03 19:32:34.631587 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.631572    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3d62efc-26ac-49a6-bedf-3babdd66abe5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd\" (UID: \"c3d62efc-26ac-49a6-bedf-3babdd66abe5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd"
Nov 03 19:32:34.631788 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.631753    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk\" (UID: \"3a1c567d-45e8-4bcb-a6a7-4f11217f2299\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk"
Nov 03 19:32:34.631890 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.631872    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk\" (UID: \"3a1c567d-45e8-4bcb-a6a7-4f11217f2299\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk"
Nov 03 19:32:34.674990 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.674957    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fzkj\" (UniqueName: \"kubernetes.io/projected/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-kube-api-access-6fzkj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk\" (UID: \"3a1c567d-45e8-4bcb-a6a7-4f11217f2299\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk"
Nov 03 19:32:34.732040 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.731966    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3d62efc-26ac-49a6-bedf-3babdd66abe5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd\" (UID: \"c3d62efc-26ac-49a6-bedf-3babdd66abe5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd"
Nov 03 19:32:34.732040 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.732004    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74c97\" (UniqueName: \"kubernetes.io/projected/c3d62efc-26ac-49a6-bedf-3babdd66abe5-kube-api-access-74c97\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd\" (UID: \"c3d62efc-26ac-49a6-bedf-3babdd66abe5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd"
Nov 03 19:32:34.732245 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.732141    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3d62efc-26ac-49a6-bedf-3babdd66abe5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd\" (UID: \"c3d62efc-26ac-49a6-bedf-3babdd66abe5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd"
Nov 03 19:32:34.732358 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.732337    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3d62efc-26ac-49a6-bedf-3babdd66abe5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd\" (UID: \"c3d62efc-26ac-49a6-bedf-3babdd66abe5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd"
Nov 03 19:32:34.732438 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.732422    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3d62efc-26ac-49a6-bedf-3babdd66abe5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd\" (UID: \"c3d62efc-26ac-49a6-bedf-3babdd66abe5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd"
Nov 03 19:32:34.744451 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.744424    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74c97\" (UniqueName: \"kubernetes.io/projected/c3d62efc-26ac-49a6-bedf-3babdd66abe5-kube-api-access-74c97\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd\" (UID: \"c3d62efc-26ac-49a6-bedf-3babdd66abe5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd"
Nov 03 19:32:34.786987 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.786956    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk"
Nov 03 19:32:34.885700 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.885672    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd"
Nov 03 19:32:34.930933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.930461    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"]
Nov 03 19:32:34.934365 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.934345    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:34.943104 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.943081    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 19:32:34.943227 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.943145    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 19:32:34.943227 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.943202    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8qmjp\""
Nov 03 19:32:34.944826 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.944806    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 19:32:34.945494 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.945465    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 19:32:34.945881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.945506    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 19:32:34.945881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.945525    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 19:32:34.949362 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.949344    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"]
Nov 03 19:32:34.988369 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:34.988298    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:35.034628 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.034594    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngk2r\" (UniqueName: \"kubernetes.io/projected/f835fba3-cf23-40c2-98d1-06435fd18080-kube-api-access-ngk2r\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.034781 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.034656    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.034781 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.034726    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.034781 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.034759    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f835fba3-cf23-40c2-98d1-06435fd18080-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.034781 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.034773    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.034990 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.034830    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f835fba3-cf23-40c2-98d1-06435fd18080-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.034990 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.034910    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f835fba3-cf23-40c2-98d1-06435fd18080-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.059754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.059724    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk"]
Nov 03 19:32:35.063364 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:32:35.063341    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1c567d_45e8_4bcb_a6a7_4f11217f2299.slice/crio-bf86972fa448de6d5a8a094a3631e310aae869727bf47700532a0c5d48b3a915 WatchSource:0}: Error finding container bf86972fa448de6d5a8a094a3631e310aae869727bf47700532a0c5d48b3a915: Status 404 returned error can't find the container with id bf86972fa448de6d5a8a094a3631e310aae869727bf47700532a0c5d48b3a915
Nov 03 19:32:35.118692 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.118662    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:35.136112 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.136080    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.136256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.136133    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.136256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.136168    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f835fba3-cf23-40c2-98d1-06435fd18080-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.136256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.136184    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.136256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.136209    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f835fba3-cf23-40c2-98d1-06435fd18080-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.136256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.136240    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f835fba3-cf23-40c2-98d1-06435fd18080-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.136666 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.136282    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngk2r\" (UniqueName: \"kubernetes.io/projected/f835fba3-cf23-40c2-98d1-06435fd18080-kube-api-access-ngk2r\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.137095 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.137071    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f835fba3-cf23-40c2-98d1-06435fd18080-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.138592 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.138568    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f835fba3-cf23-40c2-98d1-06435fd18080-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.138788 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.138767    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.138991 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.138974    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.139054 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.138980    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.148419 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.148396    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f835fba3-cf23-40c2-98d1-06435fd18080-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.148578 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.148560    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngk2r\" (UniqueName: \"kubernetes.io/projected/f835fba3-cf23-40c2-98d1-06435fd18080-kube-api-access-ngk2r\") pod \"istiod-openshift-gateway-7876bfcb74-jq87s\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.157763 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.157743    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd"]
Nov 03 19:32:35.160847 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:32:35.160817    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d62efc_26ac_49a6_bedf_3babdd66abe5.slice/crio-8428cbcbb8f4f4f060e8f28640144c5fde250a6668f7ceae77b35deb83c37911 WatchSource:0}: Error finding container 8428cbcbb8f4f4f060e8f28640144c5fde250a6668f7ceae77b35deb83c37911: Status 404 returned error can't find the container with id 8428cbcbb8f4f4f060e8f28640144c5fde250a6668f7ceae77b35deb83c37911
Nov 03 19:32:35.251524 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.251446    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:35.528973 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.528895    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"]
Nov 03 19:32:35.532437 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:32:35.532411    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf835fba3_cf23_40c2_98d1_06435fd18080.slice/crio-80a8a1b8d79b05a857314cf5b44da47ecc5e3f3fc5842858033403267e6d8c99 WatchSource:0}: Error finding container 80a8a1b8d79b05a857314cf5b44da47ecc5e3f3fc5842858033403267e6d8c99: Status 404 returned error can't find the container with id 80a8a1b8d79b05a857314cf5b44da47ecc5e3f3fc5842858033403267e6d8c99
Nov 03 19:32:35.887704 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.887664    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd" event={"ID":"c3d62efc-26ac-49a6-bedf-3babdd66abe5","Type":"ContainerStarted","Data":"8428cbcbb8f4f4f060e8f28640144c5fde250a6668f7ceae77b35deb83c37911"}
Nov 03 19:32:35.889117 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.889079    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk" event={"ID":"3a1c567d-45e8-4bcb-a6a7-4f11217f2299","Type":"ContainerStarted","Data":"bf86972fa448de6d5a8a094a3631e310aae869727bf47700532a0c5d48b3a915"}
Nov 03 19:32:35.890482 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:35.890454    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s" event={"ID":"f835fba3-cf23-40c2-98d1-06435fd18080","Type":"ContainerStarted","Data":"80a8a1b8d79b05a857314cf5b44da47ecc5e3f3fc5842858033403267e6d8c99"}
Nov 03 19:32:36.248430 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:36.248345    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:39.083898 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:39.083854    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:32:39.084232 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:39.083928    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:32:39.780398 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:39.780291    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:39.910723 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:39.910691    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s" event={"ID":"f835fba3-cf23-40c2-98d1-06435fd18080","Type":"ContainerStarted","Data":"081ad13c636149ecd1076bf7cf48a757728685b0aeb1fd80737110efce1ea543"}
Nov 03 19:32:39.910969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:39.910948    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:39.919647 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:39.919578    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:39.939371 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:39.939311    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s" podStartSLOduration=2.39072594 podStartE2EDuration="5.93929292s" totalImagesPullingTime="3.54856698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:39.938532073 +0000 UTC m=+316.192839390" watchObservedRunningTime="2026-11-03 19:32:39.93929292 +0000 UTC m=+316.193600145"
Nov 03 19:32:40.110203 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:40.110123    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:40.252010 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:40.251921    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:40.381061 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:40.380911    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:40.916870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:40.916828    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:32:41.920728 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:41.920649    5237 generic.go:356] "Generic (PLEG): container finished" podID="3a1c567d-45e8-4bcb-a6a7-4f11217f2299" containerID="145989c747ba30edf3df5de53d231dea98ab5eaa3de825f6d24b8088e9bff6b6" exitCode=0
Nov 03 19:32:41.921171 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:41.920747    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk" event={"ID":"3a1c567d-45e8-4bcb-a6a7-4f11217f2299","Type":"ContainerDied","Data":"145989c747ba30edf3df5de53d231dea98ab5eaa3de825f6d24b8088e9bff6b6"}
Nov 03 19:32:42.927121 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:42.927092    5237 generic.go:356] "Generic (PLEG): container finished" podID="c3d62efc-26ac-49a6-bedf-3babdd66abe5" containerID="9f9778aca3546b4a7cbceaee0dda1ea5f84795d05754ad836ebc444d9c3c41e2" exitCode=0
Nov 03 19:32:42.927607 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:42.927169    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd" event={"ID":"c3d62efc-26ac-49a6-bedf-3babdd66abe5","Type":"ContainerDied","Data":"9f9778aca3546b4a7cbceaee0dda1ea5f84795d05754ad836ebc444d9c3c41e2"}
Nov 03 19:32:43.054961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:43.054925    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:43.254672 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:32:43.254593    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:32:45.941925 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:45.941889    5237 generic.go:356] "Generic (PLEG): container finished" podID="c3d62efc-26ac-49a6-bedf-3babdd66abe5" containerID="39583105d3ea5face7b4c8f09dfa3436f7b0040c7d9eeeac44726ef9f7cab3e4" exitCode=0
Nov 03 19:32:45.942345 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:45.941983    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd" event={"ID":"c3d62efc-26ac-49a6-bedf-3babdd66abe5","Type":"ContainerDied","Data":"39583105d3ea5face7b4c8f09dfa3436f7b0040c7d9eeeac44726ef9f7cab3e4"}
Nov 03 19:32:45.943876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:45.943856    5237 generic.go:356] "Generic (PLEG): container finished" podID="3a1c567d-45e8-4bcb-a6a7-4f11217f2299" containerID="2c1fa82b255bd3f1a176d45b67c83844dd224f5c2ccdca0a5a1dcf477b151dd3" exitCode=0
Nov 03 19:32:45.943938 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:45.943882    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk" event={"ID":"3a1c567d-45e8-4bcb-a6a7-4f11217f2299","Type":"ContainerDied","Data":"2c1fa82b255bd3f1a176d45b67c83844dd224f5c2ccdca0a5a1dcf477b151dd3"}
Nov 03 19:32:52.973031 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:52.972942    5237 generic.go:356] "Generic (PLEG): container finished" podID="c3d62efc-26ac-49a6-bedf-3babdd66abe5" containerID="9a78907fdd6c7e9d1c9f9c6d6593b256babb9026979954049455f416d2a45ea0" exitCode=0
Nov 03 19:32:52.973031 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:52.973020    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd" event={"ID":"c3d62efc-26ac-49a6-bedf-3babdd66abe5","Type":"ContainerDied","Data":"9a78907fdd6c7e9d1c9f9c6d6593b256babb9026979954049455f416d2a45ea0"}
Nov 03 19:32:52.975029 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:52.975006    5237 generic.go:356] "Generic (PLEG): container finished" podID="3a1c567d-45e8-4bcb-a6a7-4f11217f2299" containerID="eca51faf03237f2ea972ea07ce75572efe51d955690b16659907c4bbed811b7d" exitCode=0
Nov 03 19:32:52.975137 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:52.975064    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk" event={"ID":"3a1c567d-45e8-4bcb-a6a7-4f11217f2299","Type":"ContainerDied","Data":"eca51faf03237f2ea972ea07ce75572efe51d955690b16659907c4bbed811b7d"}
Nov 03 19:32:54.099574 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.099548    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd"
Nov 03 19:32:54.128515 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.128485    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk"
Nov 03 19:32:54.208571 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.208534    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d62efc-26ac-49a6-bedf-3babdd66abe5-kube-api-access-74c97\" (UniqueName: \"kubernetes.io/projected/c3d62efc-26ac-49a6-bedf-3babdd66abe5-kube-api-access-74c97\") pod \"c3d62efc-26ac-49a6-bedf-3babdd66abe5\" (UID: \"c3d62efc-26ac-49a6-bedf-3babdd66abe5\") "
Nov 03 19:32:54.208762 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.208604    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d62efc-26ac-49a6-bedf-3babdd66abe5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3d62efc-26ac-49a6-bedf-3babdd66abe5-bundle\") pod \"c3d62efc-26ac-49a6-bedf-3babdd66abe5\" (UID: \"c3d62efc-26ac-49a6-bedf-3babdd66abe5\") "
Nov 03 19:32:54.208762 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.208629    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-kube-api-access-6fzkj\" (UniqueName: \"kubernetes.io/projected/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-kube-api-access-6fzkj\") pod \"3a1c567d-45e8-4bcb-a6a7-4f11217f2299\" (UID: \"3a1c567d-45e8-4bcb-a6a7-4f11217f2299\") "
Nov 03 19:32:54.208762 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.208654    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-bundle\") pod \"3a1c567d-45e8-4bcb-a6a7-4f11217f2299\" (UID: \"3a1c567d-45e8-4bcb-a6a7-4f11217f2299\") "
Nov 03 19:32:54.208762 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.208673    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d62efc-26ac-49a6-bedf-3babdd66abe5-util\" (UniqueName: \"kubernetes.io/empty-dir/c3d62efc-26ac-49a6-bedf-3babdd66abe5-util\") pod \"c3d62efc-26ac-49a6-bedf-3babdd66abe5\" (UID: \"c3d62efc-26ac-49a6-bedf-3babdd66abe5\") "
Nov 03 19:32:54.208762 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.208696    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-util\" (UniqueName: \"kubernetes.io/empty-dir/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-util\") pod \"3a1c567d-45e8-4bcb-a6a7-4f11217f2299\" (UID: \"3a1c567d-45e8-4bcb-a6a7-4f11217f2299\") "
Nov 03 19:32:54.209201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.209174    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-bundle" pod "3a1c567d-45e8-4bcb-a6a7-4f11217f2299" (UID: "3a1c567d-45e8-4bcb-a6a7-4f11217f2299"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:32:54.209511 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.209478    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d62efc-26ac-49a6-bedf-3babdd66abe5-bundle" pod "c3d62efc-26ac-49a6-bedf-3babdd66abe5" (UID: "c3d62efc-26ac-49a6-bedf-3babdd66abe5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:32:54.210954 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.210924    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d62efc-26ac-49a6-bedf-3babdd66abe5-kube-api-access-74c97" pod "c3d62efc-26ac-49a6-bedf-3babdd66abe5" (UID: "c3d62efc-26ac-49a6-bedf-3babdd66abe5"). InnerVolumeSpecName "kube-api-access-74c97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:54.210954 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.210934    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-kube-api-access-6fzkj" pod "3a1c567d-45e8-4bcb-a6a7-4f11217f2299" (UID: "3a1c567d-45e8-4bcb-a6a7-4f11217f2299"). InnerVolumeSpecName "kube-api-access-6fzkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:54.215675 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.215639    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d62efc-26ac-49a6-bedf-3babdd66abe5-util" pod "c3d62efc-26ac-49a6-bedf-3babdd66abe5" (UID: "c3d62efc-26ac-49a6-bedf-3babdd66abe5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:32:54.216296 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.216277    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-util" pod "3a1c567d-45e8-4bcb-a6a7-4f11217f2299" (UID: "3a1c567d-45e8-4bcb-a6a7-4f11217f2299"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:32:54.309248 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.309216    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74c97\" (UniqueName: \"kubernetes.io/projected/c3d62efc-26ac-49a6-bedf-3babdd66abe5-kube-api-access-74c97\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:54.309248 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.309247    5237 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3d62efc-26ac-49a6-bedf-3babdd66abe5-bundle\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:54.309422 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.309259    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fzkj\" (UniqueName: \"kubernetes.io/projected/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-kube-api-access-6fzkj\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:54.309422 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.309267    5237 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-bundle\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:54.309422 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.309275    5237 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3d62efc-26ac-49a6-bedf-3babdd66abe5-util\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:54.309422 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.309283    5237 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a1c567d-45e8-4bcb-a6a7-4f11217f2299-util\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:54.983725 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.983683    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd" event={"ID":"c3d62efc-26ac-49a6-bedf-3babdd66abe5","Type":"ContainerDied","Data":"8428cbcbb8f4f4f060e8f28640144c5fde250a6668f7ceae77b35deb83c37911"}
Nov 03 19:32:54.983944 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.983734    5237 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8428cbcbb8f4f4f060e8f28640144c5fde250a6668f7ceae77b35deb83c37911"
Nov 03 19:32:54.983944 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.983695    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8xkvd"
Nov 03 19:32:54.985590 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.985565    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk" event={"ID":"3a1c567d-45e8-4bcb-a6a7-4f11217f2299","Type":"ContainerDied","Data":"bf86972fa448de6d5a8a094a3631e310aae869727bf47700532a0c5d48b3a915"}
Nov 03 19:32:54.985689 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.985597    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429nzk"
Nov 03 19:32:54.985689 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:32:54.985599    5237 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf86972fa448de6d5a8a094a3631e310aae869727bf47700532a0c5d48b3a915"
Nov 03 19:33:03.694733 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:03.694696    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:03.845547 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:03.845506    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:04.556041 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.556006    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:04.666666 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.666633    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"]
Nov 03 19:33:04.666962 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.666948    5237 cpu_manager.go:414] "containerMap: removing container" podUID="3a1c567d-45e8-4bcb-a6a7-4f11217f2299" containerName="extract"
Nov 03 19:33:04.667018 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.666963    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1c567d-45e8-4bcb-a6a7-4f11217f2299" containerName="extract"
Nov 03 19:33:04.667018 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.666978    5237 cpu_manager.go:414] "containerMap: removing container" podUID="c3d62efc-26ac-49a6-bedf-3babdd66abe5" containerName="util"
Nov 03 19:33:04.667018 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.666983    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d62efc-26ac-49a6-bedf-3babdd66abe5" containerName="util"
Nov 03 19:33:04.667018 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.666993    5237 cpu_manager.go:414] "containerMap: removing container" podUID="3a1c567d-45e8-4bcb-a6a7-4f11217f2299" containerName="pull"
Nov 03 19:33:04.667018 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.666997    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1c567d-45e8-4bcb-a6a7-4f11217f2299" containerName="pull"
Nov 03 19:33:04.667018 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.667004    5237 cpu_manager.go:414] "containerMap: removing container" podUID="3a1c567d-45e8-4bcb-a6a7-4f11217f2299" containerName="util"
Nov 03 19:33:04.667018 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.667008    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1c567d-45e8-4bcb-a6a7-4f11217f2299" containerName="util"
Nov 03 19:33:04.667018 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.667019    5237 cpu_manager.go:414] "containerMap: removing container" podUID="c3d62efc-26ac-49a6-bedf-3babdd66abe5" containerName="extract"
Nov 03 19:33:04.667246 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.667023    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d62efc-26ac-49a6-bedf-3babdd66abe5" containerName="extract"
Nov 03 19:33:04.667246 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.667029    5237 cpu_manager.go:414] "containerMap: removing container" podUID="c3d62efc-26ac-49a6-bedf-3babdd66abe5" containerName="pull"
Nov 03 19:33:04.667246 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.667034    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d62efc-26ac-49a6-bedf-3babdd66abe5" containerName="pull"
Nov 03 19:33:04.667246 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.667085    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d62efc-26ac-49a6-bedf-3babdd66abe5" containerName="extract"
Nov 03 19:33:04.667246 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.667093    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1c567d-45e8-4bcb-a6a7-4f11217f2299" containerName="extract"
Nov 03 19:33:04.673453 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.673431    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.731691 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.731655    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"]
Nov 03 19:33:04.771121 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.771082    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 19:33:04.771285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.771155    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 19:33:04.771285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.771179    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-4m7l6\""
Nov 03 19:33:04.771285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.771194    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 19:33:04.771285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.771219    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 19:33:04.772384 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.772368    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 19:33:04.794945 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.794912    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c577ed1e-9a43-4837-9f01-64bbc37a71b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.794945 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.794946    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c577ed1e-9a43-4837-9f01-64bbc37a71b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.795164 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.795054    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfhgb\" (UniqueName: \"kubernetes.io/projected/c577ed1e-9a43-4837-9f01-64bbc37a71b5-kube-api-access-wfhgb\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.795164 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.795112    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c577ed1e-9a43-4837-9f01-64bbc37a71b5-metrics-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.795164 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.795155    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c577ed1e-9a43-4837-9f01-64bbc37a71b5-manifests\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.795355 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.795202    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c577ed1e-9a43-4837-9f01-64bbc37a71b5-tmp\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.896345 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.896249    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfhgb\" (UniqueName: \"kubernetes.io/projected/c577ed1e-9a43-4837-9f01-64bbc37a71b5-kube-api-access-wfhgb\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.896345 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.896293    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c577ed1e-9a43-4837-9f01-64bbc37a71b5-metrics-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.896345 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.896310    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c577ed1e-9a43-4837-9f01-64bbc37a71b5-manifests\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.896345 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.896326    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c577ed1e-9a43-4837-9f01-64bbc37a71b5-tmp\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.896674 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.896360    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c577ed1e-9a43-4837-9f01-64bbc37a71b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.896674 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.896454    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c577ed1e-9a43-4837-9f01-64bbc37a71b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.896755 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.896726    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c577ed1e-9a43-4837-9f01-64bbc37a71b5-manifests\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.898716 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.898691    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c577ed1e-9a43-4837-9f01-64bbc37a71b5-tmp\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.898937 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.898915    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c577ed1e-9a43-4837-9f01-64bbc37a71b5-metrics-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.899111 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.899091    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c577ed1e-9a43-4837-9f01-64bbc37a71b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.899154 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.899091    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c577ed1e-9a43-4837-9f01-64bbc37a71b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.909604 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.909580    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfhgb\" (UniqueName: \"kubernetes.io/projected/c577ed1e-9a43-4837-9f01-64bbc37a71b5-kube-api-access-wfhgb\") pod \"opendatahub-operator-controller-manager-7646b49899-sdczk\" (UID: \"c577ed1e-9a43-4837-9f01-64bbc37a71b5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:04.935714 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.935680    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:04.985002 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:04.984970    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:05.095880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:05.095815    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:05.279353 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:05.279271    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:05.291511 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:05.291457    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"]
Nov 03 19:33:05.297001 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:33:05.296978    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc577ed1e_9a43_4837_9f01_64bbc37a71b5.slice/crio-f7d42264e13d71e5295cb354f088cb30519800b159b8bb25dddc427351245e25 WatchSource:0}: Error finding container f7d42264e13d71e5295cb354f088cb30519800b159b8bb25dddc427351245e25: Status 404 returned error can't find the container with id f7d42264e13d71e5295cb354f088cb30519800b159b8bb25dddc427351245e25
Nov 03 19:33:05.298581 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:05.298563    5237 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:33:05.503848 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:05.503808    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:06.024248 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:06.024170    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk" event={"ID":"c577ed1e-9a43-4837-9f01-64bbc37a71b5","Type":"ContainerStarted","Data":"f7d42264e13d71e5295cb354f088cb30519800b159b8bb25dddc427351245e25"}
Nov 03 19:33:07.327214 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:07.327175    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:07.479409 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:07.479341    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:09.037593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:09.037557    5237 generic.go:356] "Generic (PLEG): container finished" podID="c577ed1e-9a43-4837-9f01-64bbc37a71b5" containerID="95e74054e8fbd6abb70e80302a32b31aa9fe772aa2ac001706f87144babcd702" exitCode=0
Nov 03 19:33:09.037962 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:09.037639    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk" event={"ID":"c577ed1e-9a43-4837-9f01-64bbc37a71b5","Type":"ContainerDied","Data":"95e74054e8fbd6abb70e80302a32b31aa9fe772aa2ac001706f87144babcd702"}
Nov 03 19:33:10.043881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:10.043828    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk" event={"ID":"c577ed1e-9a43-4837-9f01-64bbc37a71b5","Type":"ContainerStarted","Data":"055937c01635f6147a2071b56dcc2b4497b34482c1c2dc1242d6acc195f61d59"}
Nov 03 19:33:10.044363 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:10.043961    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:10.091734 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:10.091681    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk" podStartSLOduration=2.466407045 podStartE2EDuration="6.091666834s" totalImagesPullingTime="3.187630999s" totalInitContainerRuntime="437.62879ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:33:10.090083806 +0000 UTC m=+346.344391042" watchObservedRunningTime="2026-11-03 19:33:10.091666834 +0000 UTC m=+346.345974052"
Nov 03 19:33:13.254500 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:33:13.254457    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:33:13.394474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:13.394439    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:17.023202 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:17.023165    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:18.296252 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:18.296208    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:20.441916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:20.441874    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:21.050035 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:21.049999    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-sdczk"
Nov 03 19:33:21.427652 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:21.427563    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:21.772336 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:21.772248    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:22.290015 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:22.289977    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:23.067663 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:23.067633    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:24.353108 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:24.353079    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:25.688883 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:25.688780    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:26.054233 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:26.054200    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:26.243937 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:26.243897    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:26.582725 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:26.582685    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:28.389716 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:28.389682    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:28.898664 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:28.898632    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:29.180266 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:29.180179    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:29.417487 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:29.417456    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:30.139909 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:30.139880    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:31.716301 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:31.716267    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:31.925510 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:31.925473    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:33.744085 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:33.744054    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-m8n7h"]
Nov 03 19:33:33.747944 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:33.747924    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-m8n7h"
Nov 03 19:33:33.762440 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:33.762413    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-m8n7h"]
Nov 03 19:33:33.762440 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:33.762435    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 19:33:33.762636 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:33.762507    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 19:33:33.762636 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:33.762526    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-nt9m6\""
Nov 03 19:33:33.793455 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:33.793416    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfpvr\" (UniqueName: \"kubernetes.io/projected/e2402039-dd0c-47b4-be42-1edef761c1e6-kube-api-access-mfpvr\") pod \"authorino-operator-7b7bb4bc77-m8n7h\" (UID: \"e2402039-dd0c-47b4-be42-1edef761c1e6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-m8n7h"
Nov 03 19:33:33.894415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:33.894379    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfpvr\" (UniqueName: \"kubernetes.io/projected/e2402039-dd0c-47b4-be42-1edef761c1e6-kube-api-access-mfpvr\") pod \"authorino-operator-7b7bb4bc77-m8n7h\" (UID: \"e2402039-dd0c-47b4-be42-1edef761c1e6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-m8n7h"
Nov 03 19:33:33.919669 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:33.919632    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfpvr\" (UniqueName: \"kubernetes.io/projected/e2402039-dd0c-47b4-be42-1edef761c1e6-kube-api-access-mfpvr\") pod \"authorino-operator-7b7bb4bc77-m8n7h\" (UID: \"e2402039-dd0c-47b4-be42-1edef761c1e6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-m8n7h"
Nov 03 19:33:34.058456 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:34.058424    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-m8n7h"
Nov 03 19:33:34.342548 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:34.342475    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-m8n7h"]
Nov 03 19:33:34.345637 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:33:34.345605    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2402039_dd0c_47b4_be42_1edef761c1e6.slice/crio-1fd71eef4e2fc255dc51b562bfbbdf3e982cf818e0a4cc4c9e0e94d9600239f2 WatchSource:0}: Error finding container 1fd71eef4e2fc255dc51b562bfbbdf3e982cf818e0a4cc4c9e0e94d9600239f2: Status 404 returned error can't find the container with id 1fd71eef4e2fc255dc51b562bfbbdf3e982cf818e0a4cc4c9e0e94d9600239f2
Nov 03 19:33:35.133091 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:35.133057    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-m8n7h" event={"ID":"e2402039-dd0c-47b4-be42-1edef761c1e6","Type":"ContainerStarted","Data":"1fd71eef4e2fc255dc51b562bfbbdf3e982cf818e0a4cc4c9e0e94d9600239f2"}
Nov 03 19:33:35.524771 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:35.524693    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:35.688010 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:35.687976    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:35.845357 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:35.845323    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:36.430844 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.430800    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:36.643548 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.643486    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7b4fd54d5-wn7tv"]
Nov 03 19:33:36.646608 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.646590    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b4fd54d5-wn7tv"
Nov 03 19:33:36.659387 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.659368    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 19:33:36.659637 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.659623    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 19:33:36.659732 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.659631    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-86cdm\""
Nov 03 19:33:36.676522 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.676495    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b4fd54d5-wn7tv"]
Nov 03 19:33:36.718066 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.717779    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5c9e63-14b4-49b8-b8be-44639d2e8d9e-tls-certs\") pod \"kube-auth-proxy-7b4fd54d5-wn7tv\" (UID: \"3e5c9e63-14b4-49b8-b8be-44639d2e8d9e\") " pod="openshift-ingress/kube-auth-proxy-7b4fd54d5-wn7tv"
Nov 03 19:33:36.718066 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.717845    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcxbd\" (UniqueName: \"kubernetes.io/projected/3e5c9e63-14b4-49b8-b8be-44639d2e8d9e-kube-api-access-lcxbd\") pod \"kube-auth-proxy-7b4fd54d5-wn7tv\" (UID: \"3e5c9e63-14b4-49b8-b8be-44639d2e8d9e\") " pod="openshift-ingress/kube-auth-proxy-7b4fd54d5-wn7tv"
Nov 03 19:33:36.718066 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.717883    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e5c9e63-14b4-49b8-b8be-44639d2e8d9e-tmp\") pod \"kube-auth-proxy-7b4fd54d5-wn7tv\" (UID: \"3e5c9e63-14b4-49b8-b8be-44639d2e8d9e\") " pod="openshift-ingress/kube-auth-proxy-7b4fd54d5-wn7tv"
Nov 03 19:33:36.718634 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.718599    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:36.818336 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.818306    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5c9e63-14b4-49b8-b8be-44639d2e8d9e-tls-certs\") pod \"kube-auth-proxy-7b4fd54d5-wn7tv\" (UID: \"3e5c9e63-14b4-49b8-b8be-44639d2e8d9e\") " pod="openshift-ingress/kube-auth-proxy-7b4fd54d5-wn7tv"
Nov 03 19:33:36.818545 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.818343    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcxbd\" (UniqueName: \"kubernetes.io/projected/3e5c9e63-14b4-49b8-b8be-44639d2e8d9e-kube-api-access-lcxbd\") pod \"kube-auth-proxy-7b4fd54d5-wn7tv\" (UID: \"3e5c9e63-14b4-49b8-b8be-44639d2e8d9e\") " pod="openshift-ingress/kube-auth-proxy-7b4fd54d5-wn7tv"
Nov 03 19:33:36.818545 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.818374    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e5c9e63-14b4-49b8-b8be-44639d2e8d9e-tmp\") pod \"kube-auth-proxy-7b4fd54d5-wn7tv\" (UID: \"3e5c9e63-14b4-49b8-b8be-44639d2e8d9e\") " pod="openshift-ingress/kube-auth-proxy-7b4fd54d5-wn7tv"
Nov 03 19:33:36.820792 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.820751    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e5c9e63-14b4-49b8-b8be-44639d2e8d9e-tmp\") pod \"kube-auth-proxy-7b4fd54d5-wn7tv\" (UID: \"3e5c9e63-14b4-49b8-b8be-44639d2e8d9e\") " pod="openshift-ingress/kube-auth-proxy-7b4fd54d5-wn7tv"
Nov 03 19:33:36.820940 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.820856    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5c9e63-14b4-49b8-b8be-44639d2e8d9e-tls-certs\") pod \"kube-auth-proxy-7b4fd54d5-wn7tv\" (UID: \"3e5c9e63-14b4-49b8-b8be-44639d2e8d9e\") " pod="openshift-ingress/kube-auth-proxy-7b4fd54d5-wn7tv"
Nov 03 19:33:36.840769 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.840738    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcxbd\" (UniqueName: \"kubernetes.io/projected/3e5c9e63-14b4-49b8-b8be-44639d2e8d9e-kube-api-access-lcxbd\") pod \"kube-auth-proxy-7b4fd54d5-wn7tv\" (UID: \"3e5c9e63-14b4-49b8-b8be-44639d2e8d9e\") " pod="openshift-ingress/kube-auth-proxy-7b4fd54d5-wn7tv"
Nov 03 19:33:36.958918 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:36.958877    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b4fd54d5-wn7tv"
Nov 03 19:33:37.015798 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:37.015706    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:37.187890 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:37.187850    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:37.247948 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:37.247811    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b4fd54d5-wn7tv"]
Nov 03 19:33:37.251862 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:33:37.251819    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5c9e63_14b4_49b8_b8be_44639d2e8d9e.slice/crio-25fb4f0806b65222a7b91b3e16ab5d9874a02825e3a8df96f9f250377504932f WatchSource:0}: Error finding container 25fb4f0806b65222a7b91b3e16ab5d9874a02825e3a8df96f9f250377504932f: Status 404 returned error can't find the container with id 25fb4f0806b65222a7b91b3e16ab5d9874a02825e3a8df96f9f250377504932f
Nov 03 19:33:37.450358 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:37.450323    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:38.147703 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:38.147665    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b4fd54d5-wn7tv" event={"ID":"3e5c9e63-14b4-49b8-b8be-44639d2e8d9e","Type":"ContainerStarted","Data":"25fb4f0806b65222a7b91b3e16ab5d9874a02825e3a8df96f9f250377504932f"}
Nov 03 19:33:38.463346 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:38.463245    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:40.079096 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:40.079012    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:40.157792 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:40.157760    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b4fd54d5-wn7tv" event={"ID":"3e5c9e63-14b4-49b8-b8be-44639d2e8d9e","Type":"ContainerStarted","Data":"1ee04d35fe8176ae5d65e60abd06f4d6141e2c3c202928d822251ac78c42bd93"}
Nov 03 19:33:40.362165 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:40.362090    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:41.528644 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:41.528603    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:42.060590 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.060538    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:42.529456 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.529330    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7b4fd54d5-wn7tv" podStartSLOduration=3.97387969 podStartE2EDuration="6.529303807s" totalImagesPullingTime="2.555424117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:33:40.210822941 +0000 UTC m=+376.465130166" watchObservedRunningTime="2026-11-03 19:33:42.529303807 +0000 UTC m=+378.783611090"
Nov 03 19:33:42.530771 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.530745    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-csmwj"]
Nov 03 19:33:42.536861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.536825    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csmwj"
Nov 03 19:33:42.547522 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.547497    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-csmwj"]
Nov 03 19:33:42.548014 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.547989    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 19:33:42.548129 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.548042    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-frwwl\""
Nov 03 19:33:42.548129 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.548068    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 19:33:42.569071 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.569037    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4376e17-400a-4487-bdff-ef9878f40a0e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-csmwj\" (UID: \"c4376e17-400a-4487-bdff-ef9878f40a0e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csmwj"
Nov 03 19:33:42.569248 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.569102    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c4376e17-400a-4487-bdff-ef9878f40a0e-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-csmwj\" (UID: \"c4376e17-400a-4487-bdff-ef9878f40a0e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csmwj"
Nov 03 19:33:42.569248 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.569202    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czlt9\" (UniqueName: \"kubernetes.io/projected/c4376e17-400a-4487-bdff-ef9878f40a0e-kube-api-access-czlt9\") pod \"kuadrant-console-plugin-66c4fb575d-csmwj\" (UID: \"c4376e17-400a-4487-bdff-ef9878f40a0e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csmwj"
Nov 03 19:33:42.604853 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.604803    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:42.670359 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.670319    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4376e17-400a-4487-bdff-ef9878f40a0e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-csmwj\" (UID: \"c4376e17-400a-4487-bdff-ef9878f40a0e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csmwj"
Nov 03 19:33:42.670541 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.670386    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c4376e17-400a-4487-bdff-ef9878f40a0e-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-csmwj\" (UID: \"c4376e17-400a-4487-bdff-ef9878f40a0e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csmwj"
Nov 03 19:33:42.670541 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.670416    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czlt9\" (UniqueName: \"kubernetes.io/projected/c4376e17-400a-4487-bdff-ef9878f40a0e-kube-api-access-czlt9\") pod \"kuadrant-console-plugin-66c4fb575d-csmwj\" (UID: \"c4376e17-400a-4487-bdff-ef9878f40a0e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csmwj"
Nov 03 19:33:42.671195 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.671167    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c4376e17-400a-4487-bdff-ef9878f40a0e-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-csmwj\" (UID: \"c4376e17-400a-4487-bdff-ef9878f40a0e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csmwj"
Nov 03 19:33:42.673008 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.672984    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4376e17-400a-4487-bdff-ef9878f40a0e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-csmwj\" (UID: \"c4376e17-400a-4487-bdff-ef9878f40a0e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csmwj"
Nov 03 19:33:42.684742 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.684709    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czlt9\" (UniqueName: \"kubernetes.io/projected/c4376e17-400a-4487-bdff-ef9878f40a0e-kube-api-access-czlt9\") pod \"kuadrant-console-plugin-66c4fb575d-csmwj\" (UID: \"c4376e17-400a-4487-bdff-ef9878f40a0e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csmwj"
Nov 03 19:33:42.745152 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.745116    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:42.855758 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:42.855721    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csmwj"
Nov 03 19:33:43.127718 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:43.127638    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-csmwj"]
Nov 03 19:33:43.131342 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:33:43.131312    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4376e17_400a_4487_bdff_ef9878f40a0e.slice/crio-4f58e74ec82a97f00805cda2e7d51bbfc2ead41251b208e04d1d301ee2034e38 WatchSource:0}: Error finding container 4f58e74ec82a97f00805cda2e7d51bbfc2ead41251b208e04d1d301ee2034e38: Status 404 returned error can't find the container with id 4f58e74ec82a97f00805cda2e7d51bbfc2ead41251b208e04d1d301ee2034e38
Nov 03 19:33:43.170660 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:43.170628    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csmwj" event={"ID":"c4376e17-400a-4487-bdff-ef9878f40a0e","Type":"ContainerStarted","Data":"4f58e74ec82a97f00805cda2e7d51bbfc2ead41251b208e04d1d301ee2034e38"}
Nov 03 19:33:43.231748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:43.231712    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:43.257689 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:33:43.257652    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:33:45.597573 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:45.597483    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:45.748937 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:45.748893    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:45.888325 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:45.888242    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:46.055987 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:46.055947    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:46.292693 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:46.292653    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:49.196927 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:49.196892    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csmwj" event={"ID":"c4376e17-400a-4487-bdff-ef9878f40a0e","Type":"ContainerStarted","Data":"48ec7e8c49405e254cda0afc64c44dee5bc5563dc6109003197ff6692178a0e3"}
Nov 03 19:33:49.216574 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:49.216517    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csmwj" podStartSLOduration=1.309466892 podStartE2EDuration="7.216502537s" totalImagesPullingTime="5.907035645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:33:49.214699483 +0000 UTC m=+385.469006702" watchObservedRunningTime="2026-11-03 19:33:49.216502537 +0000 UTC m=+385.470810141"
Nov 03 19:33:53.724872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:53.724827    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:53.900681 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:53.900638    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:54.354536 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:54.354499    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:54.494038 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:54.494001    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:55.222493 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:55.222461    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-m8n7h" event={"ID":"e2402039-dd0c-47b4-be42-1edef761c1e6","Type":"ContainerStarted","Data":"3303bf2dd85d2f425d99be85b7ffb315f6de961d767f5591bdac9dfc9a03ec12"}
Nov 03 19:33:55.222913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:55.222556    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-m8n7h"
Nov 03 19:33:55.243096 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:55.243044    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-m8n7h" podStartSLOduration=2.015988649 podStartE2EDuration="22.243030126s" totalImagesPullingTime="20.227041477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:33:55.241523914 +0000 UTC m=+391.495831140" watchObservedRunningTime="2026-11-03 19:33:55.243030126 +0000 UTC m=+391.497337350"
Nov 03 19:33:56.927497 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:56.927462    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:57.167029 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:57.166991    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:57.934976 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:57.934934    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:58.075882 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:58.075825    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:58.713084 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:58.713043    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:58.843448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:33:58.843411    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:00.110353 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:00.110307    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:00.247957 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:00.247914    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:04.563881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:04.563812    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:04.702599 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:04.702568    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:04.833346 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:04.833265    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:05.118381 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:05.118296    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:05.249534 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:05.249496    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:05.438430 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:05.438338    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:06.228999 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:06.228972    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-m8n7h"
Nov 03 19:34:09.213969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:09.213933    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:09.392250 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:09.392212    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:13.254892 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:34:13.254850    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:34:14.906318 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:14.906278    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:15.037542 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:15.037507    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:15.315658 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:15.315618    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:15.445099 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:15.445056    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:23.789425 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:23.789384    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:23.924302 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:23.924262    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:25.062708 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.062667    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:25.089484 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.089455    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-84p5p"]
Nov 03 19:34:25.111285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.111255    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-84p5p"
Nov 03 19:34:25.123677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.123654    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wgc2d\""
Nov 03 19:34:25.123677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.123668    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 19:34:25.123677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.123660    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 19:34:25.124533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.124514    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-84p5p"]
Nov 03 19:34:25.128201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.128177    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 19:34:25.128201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.128201    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 19:34:25.128338 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.128203    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 19:34:25.212091 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.212056    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:25.229183 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.229154    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/856f0eec-2077-45e2-8ba2-fab953e8e56a-metrics-certs\") pod \"maas-controller-579cf7c6d-84p5p\" (UID: \"856f0eec-2077-45e2-8ba2-fab953e8e56a\") " pod="opendatahub/maas-controller-579cf7c6d-84p5p"
Nov 03 19:34:25.229334 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.229194    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8gdh\" (UniqueName: \"kubernetes.io/projected/856f0eec-2077-45e2-8ba2-fab953e8e56a-kube-api-access-b8gdh\") pod \"maas-controller-579cf7c6d-84p5p\" (UID: \"856f0eec-2077-45e2-8ba2-fab953e8e56a\") " pod="opendatahub/maas-controller-579cf7c6d-84p5p"
Nov 03 19:34:25.229334 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.229223    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/856f0eec-2077-45e2-8ba2-fab953e8e56a-webhook-cert\") pod \"maas-controller-579cf7c6d-84p5p\" (UID: \"856f0eec-2077-45e2-8ba2-fab953e8e56a\") " pod="opendatahub/maas-controller-579cf7c6d-84p5p"
Nov 03 19:34:25.330468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.330386    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/856f0eec-2077-45e2-8ba2-fab953e8e56a-metrics-certs\") pod \"maas-controller-579cf7c6d-84p5p\" (UID: \"856f0eec-2077-45e2-8ba2-fab953e8e56a\") " pod="opendatahub/maas-controller-579cf7c6d-84p5p"
Nov 03 19:34:25.330468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.330429    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gdh\" (UniqueName: \"kubernetes.io/projected/856f0eec-2077-45e2-8ba2-fab953e8e56a-kube-api-access-b8gdh\") pod \"maas-controller-579cf7c6d-84p5p\" (UID: \"856f0eec-2077-45e2-8ba2-fab953e8e56a\") " pod="opendatahub/maas-controller-579cf7c6d-84p5p"
Nov 03 19:34:25.330468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.330450    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/856f0eec-2077-45e2-8ba2-fab953e8e56a-webhook-cert\") pod \"maas-controller-579cf7c6d-84p5p\" (UID: \"856f0eec-2077-45e2-8ba2-fab953e8e56a\") " pod="opendatahub/maas-controller-579cf7c6d-84p5p"
Nov 03 19:34:25.332827 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.332798    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/856f0eec-2077-45e2-8ba2-fab953e8e56a-metrics-certs\") pod \"maas-controller-579cf7c6d-84p5p\" (UID: \"856f0eec-2077-45e2-8ba2-fab953e8e56a\") " pod="opendatahub/maas-controller-579cf7c6d-84p5p"
Nov 03 19:34:25.332986 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.332961    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/856f0eec-2077-45e2-8ba2-fab953e8e56a-webhook-cert\") pod \"maas-controller-579cf7c6d-84p5p\" (UID: \"856f0eec-2077-45e2-8ba2-fab953e8e56a\") " pod="opendatahub/maas-controller-579cf7c6d-84p5p"
Nov 03 19:34:25.343902 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.343876    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8gdh\" (UniqueName: \"kubernetes.io/projected/856f0eec-2077-45e2-8ba2-fab953e8e56a-kube-api-access-b8gdh\") pod \"maas-controller-579cf7c6d-84p5p\" (UID: \"856f0eec-2077-45e2-8ba2-fab953e8e56a\") " pod="opendatahub/maas-controller-579cf7c6d-84p5p"
Nov 03 19:34:25.365970 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.365936    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:25.424429 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.424392    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-84p5p"
Nov 03 19:34:25.457421 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.457390    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-694f8bb88d-22dtv"]
Nov 03 19:34:25.461024 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.461002    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694f8bb88d-22dtv"
Nov 03 19:34:25.473785 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.473758    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-694f8bb88d-22dtv"]
Nov 03 19:34:25.632503 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.632400    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-webhook-cert\") pod \"maas-controller-694f8bb88d-22dtv\" (UID: \"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd\") " pod="opendatahub/maas-controller-694f8bb88d-22dtv"
Nov 03 19:34:25.632685 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.632536    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjnm9\" (UniqueName: \"kubernetes.io/projected/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-kube-api-access-zjnm9\") pod \"maas-controller-694f8bb88d-22dtv\" (UID: \"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd\") " pod="opendatahub/maas-controller-694f8bb88d-22dtv"
Nov 03 19:34:25.632685 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.632575    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-metrics-certs\") pod \"maas-controller-694f8bb88d-22dtv\" (UID: \"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd\") " pod="opendatahub/maas-controller-694f8bb88d-22dtv"
Nov 03 19:34:25.709095 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.709059    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-84p5p"]
Nov 03 19:34:25.712344 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:34:25.712318    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856f0eec_2077_45e2_8ba2_fab953e8e56a.slice/crio-0dd174895f8331dcb015b761e1dd64ef7375d31f974da04ab08995187340b6e3 WatchSource:0}: Error finding container 0dd174895f8331dcb015b761e1dd64ef7375d31f974da04ab08995187340b6e3: Status 404 returned error can't find the container with id 0dd174895f8331dcb015b761e1dd64ef7375d31f974da04ab08995187340b6e3
Nov 03 19:34:25.733581 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.733552    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjnm9\" (UniqueName: \"kubernetes.io/projected/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-kube-api-access-zjnm9\") pod \"maas-controller-694f8bb88d-22dtv\" (UID: \"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd\") " pod="opendatahub/maas-controller-694f8bb88d-22dtv"
Nov 03 19:34:25.733721 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.733590    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-metrics-certs\") pod \"maas-controller-694f8bb88d-22dtv\" (UID: \"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd\") " pod="opendatahub/maas-controller-694f8bb88d-22dtv"
Nov 03 19:34:25.733721 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.733609    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-webhook-cert\") pod \"maas-controller-694f8bb88d-22dtv\" (UID: \"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd\") " pod="opendatahub/maas-controller-694f8bb88d-22dtv"
Nov 03 19:34:25.736024 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.736003    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-webhook-cert\") pod \"maas-controller-694f8bb88d-22dtv\" (UID: \"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd\") " pod="opendatahub/maas-controller-694f8bb88d-22dtv"
Nov 03 19:34:25.736115 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.736002    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-metrics-certs\") pod \"maas-controller-694f8bb88d-22dtv\" (UID: \"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd\") " pod="opendatahub/maas-controller-694f8bb88d-22dtv"
Nov 03 19:34:25.746301 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.746274    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjnm9\" (UniqueName: \"kubernetes.io/projected/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-kube-api-access-zjnm9\") pod \"maas-controller-694f8bb88d-22dtv\" (UID: \"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd\") " pod="opendatahub/maas-controller-694f8bb88d-22dtv"
Nov 03 19:34:25.776717 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:25.776687    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694f8bb88d-22dtv"
Nov 03 19:34:26.057994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:26.057959    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-694f8bb88d-22dtv"]
Nov 03 19:34:26.060969 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:34:26.060939    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d73f5ff_57e8_4ed0_bc2c_e00ee280a7cd.slice/crio-2c5e7f026b79aea0767d644ca4a3ad31e54c122cd7ee79798ce7b33f895dd83b WatchSource:0}: Error finding container 2c5e7f026b79aea0767d644ca4a3ad31e54c122cd7ee79798ce7b33f895dd83b: Status 404 returned error can't find the container with id 2c5e7f026b79aea0767d644ca4a3ad31e54c122cd7ee79798ce7b33f895dd83b
Nov 03 19:34:26.342760 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:26.342668    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694f8bb88d-22dtv" event={"ID":"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd","Type":"ContainerStarted","Data":"2c5e7f026b79aea0767d644ca4a3ad31e54c122cd7ee79798ce7b33f895dd83b"}
Nov 03 19:34:26.344087 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:26.344044    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-84p5p" event={"ID":"856f0eec-2077-45e2-8ba2-fab953e8e56a","Type":"ContainerStarted","Data":"0dd174895f8331dcb015b761e1dd64ef7375d31f974da04ab08995187340b6e3"}
Nov 03 19:34:28.856325 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:28.856291    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:29.224398 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:29.224303    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:29.360183 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:29.360142    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-84p5p" event={"ID":"856f0eec-2077-45e2-8ba2-fab953e8e56a","Type":"ContainerStarted","Data":"9a7d9cdfe48d9d8f12e20a6763c0c3fa02b413cd08185d5ff5a8ffd73af406c5"}
Nov 03 19:34:29.360356 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:29.360253    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-84p5p"
Nov 03 19:34:29.361951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:29.361916    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694f8bb88d-22dtv" event={"ID":"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd","Type":"ContainerStarted","Data":"37d885d26711fb23c48aaad791c3841aa8d7706502786aede42d8a36631f817f"}
Nov 03 19:34:29.362074 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:29.362053    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-694f8bb88d-22dtv"
Nov 03 19:34:29.404933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:29.404877    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-84p5p" podStartSLOduration=1.280827745 podStartE2EDuration="4.404861348s" totalImagesPullingTime="3.124033603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:34:29.404287958 +0000 UTC m=+425.658595195" watchObservedRunningTime="2026-11-03 19:34:29.404861348 +0000 UTC m=+425.659168567"
Nov 03 19:34:29.456873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:29.456779    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-694f8bb88d-22dtv" podStartSLOduration=1.6319170170000001 podStartE2EDuration="4.45676391s" totalImagesPullingTime="2.824846893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:34:29.45596933 +0000 UTC m=+425.710276556" watchObservedRunningTime="2026-11-03 19:34:29.45676391 +0000 UTC m=+425.711071135"
Nov 03 19:34:29.578040 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:29.578005    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:30.584947 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.584911    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"]
Nov 03 19:34:30.588334 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.588318    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"
Nov 03 19:34:30.618191 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.618159    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"]
Nov 03 19:34:30.623225 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.623174    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 19:34:30.623225 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.623185    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 19:34:30.623393 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.623308    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-d2crb\""
Nov 03 19:34:30.623393 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.623364    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 19:34:30.670501 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.670465    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f816ed7b-55b4-4f23-9fd7-28851ef00f1c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4gb64\" (UID: \"f816ed7b-55b4-4f23-9fd7-28851ef00f1c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"
Nov 03 19:34:30.670501 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.670505    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhgvg\" (UniqueName: \"kubernetes.io/projected/f816ed7b-55b4-4f23-9fd7-28851ef00f1c-kube-api-access-mhgvg\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4gb64\" (UID: \"f816ed7b-55b4-4f23-9fd7-28851ef00f1c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"
Nov 03 19:34:30.670754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.670526    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f816ed7b-55b4-4f23-9fd7-28851ef00f1c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4gb64\" (UID: \"f816ed7b-55b4-4f23-9fd7-28851ef00f1c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"
Nov 03 19:34:30.670754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.670670    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f816ed7b-55b4-4f23-9fd7-28851ef00f1c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4gb64\" (UID: \"f816ed7b-55b4-4f23-9fd7-28851ef00f1c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"
Nov 03 19:34:30.771277 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.771237    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhgvg\" (UniqueName: \"kubernetes.io/projected/f816ed7b-55b4-4f23-9fd7-28851ef00f1c-kube-api-access-mhgvg\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4gb64\" (UID: \"f816ed7b-55b4-4f23-9fd7-28851ef00f1c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"
Nov 03 19:34:30.771277 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.771278    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f816ed7b-55b4-4f23-9fd7-28851ef00f1c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4gb64\" (UID: \"f816ed7b-55b4-4f23-9fd7-28851ef00f1c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"
Nov 03 19:34:30.771534 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.771339    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f816ed7b-55b4-4f23-9fd7-28851ef00f1c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4gb64\" (UID: \"f816ed7b-55b4-4f23-9fd7-28851ef00f1c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"
Nov 03 19:34:30.771534 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.771370    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f816ed7b-55b4-4f23-9fd7-28851ef00f1c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4gb64\" (UID: \"f816ed7b-55b4-4f23-9fd7-28851ef00f1c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"
Nov 03 19:34:30.771915 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.771893    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f816ed7b-55b4-4f23-9fd7-28851ef00f1c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4gb64\" (UID: \"f816ed7b-55b4-4f23-9fd7-28851ef00f1c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"
Nov 03 19:34:30.772247 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.772222    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f816ed7b-55b4-4f23-9fd7-28851ef00f1c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4gb64\" (UID: \"f816ed7b-55b4-4f23-9fd7-28851ef00f1c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"
Nov 03 19:34:30.773681 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.773663    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f816ed7b-55b4-4f23-9fd7-28851ef00f1c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4gb64\" (UID: \"f816ed7b-55b4-4f23-9fd7-28851ef00f1c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"
Nov 03 19:34:30.799283 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.799253    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhgvg\" (UniqueName: \"kubernetes.io/projected/f816ed7b-55b4-4f23-9fd7-28851ef00f1c-kube-api-access-mhgvg\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4gb64\" (UID: \"f816ed7b-55b4-4f23-9fd7-28851ef00f1c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"
Nov 03 19:34:30.867350 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.867267    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:30.900274 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:30.900231    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"
Nov 03 19:34:31.075464 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:31.075430    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:31.192622 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:31.192541    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"]
Nov 03 19:34:31.196154 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:34:31.196129    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf816ed7b_55b4_4f23_9fd7_28851ef00f1c.slice/crio-31fce1f6399aef539c55bd4bcefee0ba668cb87f91c34adafd00a73352dc46b6 WatchSource:0}: Error finding container 31fce1f6399aef539c55bd4bcefee0ba668cb87f91c34adafd00a73352dc46b6: Status 404 returned error can't find the container with id 31fce1f6399aef539c55bd4bcefee0ba668cb87f91c34adafd00a73352dc46b6
Nov 03 19:34:31.370040 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:31.370001    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64" event={"ID":"f816ed7b-55b4-4f23-9fd7-28851ef00f1c","Type":"ContainerStarted","Data":"31fce1f6399aef539c55bd4bcefee0ba668cb87f91c34adafd00a73352dc46b6"}
Nov 03 19:34:36.418066 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.418029    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:36.609813 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.609781    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:36.638258 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.638231    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"]
Nov 03 19:34:36.643592 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.643561    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"
Nov 03 19:34:36.670159 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.670039    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"]
Nov 03 19:34:36.670936 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.670911    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kpbwm\""
Nov 03 19:34:36.671086 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.671069    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 19:34:36.694317 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.694288    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-78kwl"]
Nov 03 19:34:36.699428 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.699408    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-78kwl"
Nov 03 19:34:36.719675 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.719641    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6vhk\" (UniqueName: \"kubernetes.io/projected/f4b0fa79-e38f-4966-9c9f-e070a4091196-kube-api-access-f6vhk\") pod \"model-serving-api-75bd89fdfc-78kwl\" (UID: \"f4b0fa79-e38f-4966-9c9f-e070a4091196\") " pod="opendatahub/model-serving-api-75bd89fdfc-78kwl"
Nov 03 19:34:36.719826 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.719753    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvk5b\" (UniqueName: \"kubernetes.io/projected/d2e6c2a2-f9b1-4505-a970-d20a42adf795-kube-api-access-fvk5b\") pod \"llmisvc-controller-manager-869d5dcc5d-h28gk\" (UID: \"d2e6c2a2-f9b1-4505-a970-d20a42adf795\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"
Nov 03 19:34:36.719826 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.719793    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b0fa79-e38f-4966-9c9f-e070a4091196-tls-certs\") pod \"model-serving-api-75bd89fdfc-78kwl\" (UID: \"f4b0fa79-e38f-4966-9c9f-e070a4091196\") " pod="opendatahub/model-serving-api-75bd89fdfc-78kwl"
Nov 03 19:34:36.719979 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.719829    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2e6c2a2-f9b1-4505-a970-d20a42adf795-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-h28gk\" (UID: \"d2e6c2a2-f9b1-4505-a970-d20a42adf795\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"
Nov 03 19:34:36.722044 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.722021    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 19:34:36.722193 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.722099    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-cc2tz\""
Nov 03 19:34:36.727376 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.727313    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-78kwl"]
Nov 03 19:34:36.817908 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.817862    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:36.821097 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.821067    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6vhk\" (UniqueName: \"kubernetes.io/projected/f4b0fa79-e38f-4966-9c9f-e070a4091196-kube-api-access-f6vhk\") pod \"model-serving-api-75bd89fdfc-78kwl\" (UID: \"f4b0fa79-e38f-4966-9c9f-e070a4091196\") " pod="opendatahub/model-serving-api-75bd89fdfc-78kwl"
Nov 03 19:34:36.821235 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.821118    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvk5b\" (UniqueName: \"kubernetes.io/projected/d2e6c2a2-f9b1-4505-a970-d20a42adf795-kube-api-access-fvk5b\") pod \"llmisvc-controller-manager-869d5dcc5d-h28gk\" (UID: \"d2e6c2a2-f9b1-4505-a970-d20a42adf795\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"
Nov 03 19:34:36.821235 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.821149    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b0fa79-e38f-4966-9c9f-e070a4091196-tls-certs\") pod \"model-serving-api-75bd89fdfc-78kwl\" (UID: \"f4b0fa79-e38f-4966-9c9f-e070a4091196\") " pod="opendatahub/model-serving-api-75bd89fdfc-78kwl"
Nov 03 19:34:36.821235 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.821183    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2e6c2a2-f9b1-4505-a970-d20a42adf795-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-h28gk\" (UID: \"d2e6c2a2-f9b1-4505-a970-d20a42adf795\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"
Nov 03 19:34:36.824604 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.824577    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2e6c2a2-f9b1-4505-a970-d20a42adf795-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-h28gk\" (UID: \"d2e6c2a2-f9b1-4505-a970-d20a42adf795\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"
Nov 03 19:34:36.826902 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.826877    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b0fa79-e38f-4966-9c9f-e070a4091196-tls-certs\") pod \"model-serving-api-75bd89fdfc-78kwl\" (UID: \"f4b0fa79-e38f-4966-9c9f-e070a4091196\") " pod="opendatahub/model-serving-api-75bd89fdfc-78kwl"
Nov 03 19:34:36.837086 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.837067    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvk5b\" (UniqueName: \"kubernetes.io/projected/d2e6c2a2-f9b1-4505-a970-d20a42adf795-kube-api-access-fvk5b\") pod \"llmisvc-controller-manager-869d5dcc5d-h28gk\" (UID: \"d2e6c2a2-f9b1-4505-a970-d20a42adf795\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"
Nov 03 19:34:36.837960 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.837937    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6vhk\" (UniqueName: \"kubernetes.io/projected/f4b0fa79-e38f-4966-9c9f-e070a4091196-kube-api-access-f6vhk\") pod \"model-serving-api-75bd89fdfc-78kwl\" (UID: \"f4b0fa79-e38f-4966-9c9f-e070a4091196\") " pod="opendatahub/model-serving-api-75bd89fdfc-78kwl"
Nov 03 19:34:36.960321 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.960253    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"
Nov 03 19:34:36.964927 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:36.964900    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:37.017178 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:37.017148    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-78kwl"
Nov 03 19:34:37.242370 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:37.242294    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"]
Nov 03 19:34:37.245436 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:34:37.245407    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd2e6c2a2_f9b1_4505_a970_d20a42adf795.slice/crio-8a56e31ced81ab3e869eeccf122a2d7bb93bcaf69ea58476a65fd41b5ebbe1f1 WatchSource:0}: Error finding container 8a56e31ced81ab3e869eeccf122a2d7bb93bcaf69ea58476a65fd41b5ebbe1f1: Status 404 returned error can't find the container with id 8a56e31ced81ab3e869eeccf122a2d7bb93bcaf69ea58476a65fd41b5ebbe1f1
Nov 03 19:34:37.342037 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:34:37.342002    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b0fa79_e38f_4966_9c9f_e070a4091196.slice/crio-58d880709b88a746ebead7b46484a7ea18069d32b6f4ce71d85f8cfa47ffc36b WatchSource:0}: Error finding container 58d880709b88a746ebead7b46484a7ea18069d32b6f4ce71d85f8cfa47ffc36b: Status 404 returned error can't find the container with id 58d880709b88a746ebead7b46484a7ea18069d32b6f4ce71d85f8cfa47ffc36b
Nov 03 19:34:37.367926 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:37.367895    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-78kwl"]
Nov 03 19:34:37.401661 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:37.401626    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-78kwl" event={"ID":"f4b0fa79-e38f-4966-9c9f-e070a4091196","Type":"ContainerStarted","Data":"58d880709b88a746ebead7b46484a7ea18069d32b6f4ce71d85f8cfa47ffc36b"}
Nov 03 19:34:37.403075 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:37.403045    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk" event={"ID":"d2e6c2a2-f9b1-4505-a970-d20a42adf795","Type":"ContainerStarted","Data":"8a56e31ced81ab3e869eeccf122a2d7bb93bcaf69ea58476a65fd41b5ebbe1f1"}
Nov 03 19:34:38.429077 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.428956    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:38.618925 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.618887    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:38.742499 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.742417    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"]
Nov 03 19:34:38.746419 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.746393    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.780274 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.780222    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"]
Nov 03 19:34:38.840827 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.840775    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7abe1b7-d7bc-402e-a762-e35a95520eb3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.841043 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.840863    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7abe1b7-d7bc-402e-a762-e35a95520eb3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.841043 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.840926    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7abe1b7-d7bc-402e-a762-e35a95520eb3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.841043 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.840954    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7abe1b7-d7bc-402e-a762-e35a95520eb3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.841043 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.840976    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7abe1b7-d7bc-402e-a762-e35a95520eb3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.841043 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.841010    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7abe1b7-d7bc-402e-a762-e35a95520eb3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.841043 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.841038    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99z7m\" (UniqueName: \"kubernetes.io/projected/f7abe1b7-d7bc-402e-a762-e35a95520eb3-kube-api-access-99z7m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.942330 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.942286    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7abe1b7-d7bc-402e-a762-e35a95520eb3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.942517 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.942349    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7abe1b7-d7bc-402e-a762-e35a95520eb3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.942517 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.942374    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7abe1b7-d7bc-402e-a762-e35a95520eb3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.942517 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.942391    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7abe1b7-d7bc-402e-a762-e35a95520eb3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.942517 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.942435    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7abe1b7-d7bc-402e-a762-e35a95520eb3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.942517 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.942472    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99z7m\" (UniqueName: \"kubernetes.io/projected/f7abe1b7-d7bc-402e-a762-e35a95520eb3-kube-api-access-99z7m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.942517 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.942508    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7abe1b7-d7bc-402e-a762-e35a95520eb3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.943475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.943442    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7abe1b7-d7bc-402e-a762-e35a95520eb3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.945952 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.945900    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7abe1b7-d7bc-402e-a762-e35a95520eb3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.946086 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.946062    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7abe1b7-d7bc-402e-a762-e35a95520eb3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.948191 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.948164    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7abe1b7-d7bc-402e-a762-e35a95520eb3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.948830 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.948782    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7abe1b7-d7bc-402e-a762-e35a95520eb3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.958432 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.958400    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99z7m\" (UniqueName: \"kubernetes.io/projected/f7abe1b7-d7bc-402e-a762-e35a95520eb3-kube-api-access-99z7m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:38.960158 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:38.960109    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7abe1b7-d7bc-402e-a762-e35a95520eb3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2prbp\" (UID: \"f7abe1b7-d7bc-402e-a762-e35a95520eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:39.061638 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:39.061605    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:39.365824 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:39.365768    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"]
Nov 03 19:34:39.372337 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:39.372298    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:34:39.372438 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:39.372363    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:34:39.417854 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:39.417802    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp" event={"ID":"f7abe1b7-d7bc-402e-a762-e35a95520eb3","Type":"ContainerStarted","Data":"d7a9dcdec27ca57db8654251648824d72323b63f31a97d6c077736e2d5174cea"}
Nov 03 19:34:40.367469 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.367388    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-694f8bb88d-22dtv"
Nov 03 19:34:40.367871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.367477    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-84p5p"
Nov 03 19:34:40.424587 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.424546    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-78kwl" event={"ID":"f4b0fa79-e38f-4966-9c9f-e070a4091196","Type":"ContainerStarted","Data":"e554b8d5ca1e5353c7f41c9a3f09919e434ec6fee15bde92e2e5d838bb26824f"}
Nov 03 19:34:40.424751 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.424643    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-78kwl"
Nov 03 19:34:40.426458 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.426428    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk" event={"ID":"d2e6c2a2-f9b1-4505-a970-d20a42adf795","Type":"ContainerStarted","Data":"69c77f11b793e2f6fe90de9b3f128ceade3d1afe61635ceed9494737a27233dd"}
Nov 03 19:34:40.426605 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.426545    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"
Nov 03 19:34:40.428281 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.428252    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp" event={"ID":"f7abe1b7-d7bc-402e-a762-e35a95520eb3","Type":"ContainerStarted","Data":"5e0b8dc5b947dd0103b3db8874aa82f77609c4b679d6d3e0355c33733e9c6754"}
Nov 03 19:34:40.428433 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.428418    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:40.430073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.430050    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp"
Nov 03 19:34:40.451887 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.451814    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-78kwl" podStartSLOduration=2.035355357 podStartE2EDuration="4.451797442s" totalImagesPullingTime="2.416442085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:34:40.44967217 +0000 UTC m=+436.703979397" watchObservedRunningTime="2026-11-03 19:34:40.451797442 +0000 UTC m=+436.706104669"
Nov 03 19:34:40.455760 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.455736    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-84p5p"]
Nov 03 19:34:40.455925 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.455905    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-84p5p" podUID="856f0eec-2077-45e2-8ba2-fab953e8e56a" containerName="manager" containerID="cri-o://9a7d9cdfe48d9d8f12e20a6763c0c3fa02b413cd08185d5ff5a8ffd73af406c5" gracePeriod=30
Nov 03 19:34:40.497124 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.497065    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2prbp" podStartSLOduration=2.497045772 podStartE2EDuration="2.497045772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:34:40.496156686 +0000 UTC m=+436.750463954" watchObservedRunningTime="2026-11-03 19:34:40.497045772 +0000 UTC m=+436.751352997"
Nov 03 19:34:40.542882 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.542817    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk" podStartSLOduration=1.706000065 podStartE2EDuration="4.542801864s" totalImagesPullingTime="2.836801799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:34:40.522870888 +0000 UTC m=+436.777178115" watchObservedRunningTime="2026-11-03 19:34:40.542801864 +0000 UTC m=+436.797109089"
Nov 03 19:34:40.543425 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.543410    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"]
Nov 03 19:34:40.543599 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.543582    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s" podUID="f835fba3-cf23-40c2-98d1-06435fd18080" containerName="discovery" containerID="cri-o://081ad13c636149ecd1076bf7cf48a757728685b0aeb1fd80737110efce1ea543" gracePeriod=30
Nov 03 19:34:40.715212 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.715187    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-84p5p"
Nov 03 19:34:40.782065 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.782031    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856f0eec-2077-45e2-8ba2-fab953e8e56a-kube-api-access-b8gdh\" (UniqueName: \"kubernetes.io/projected/856f0eec-2077-45e2-8ba2-fab953e8e56a-kube-api-access-b8gdh\") pod \"856f0eec-2077-45e2-8ba2-fab953e8e56a\" (UID: \"856f0eec-2077-45e2-8ba2-fab953e8e56a\") "
Nov 03 19:34:40.782255 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.782086    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/856f0eec-2077-45e2-8ba2-fab953e8e56a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/856f0eec-2077-45e2-8ba2-fab953e8e56a-metrics-certs\") pod \"856f0eec-2077-45e2-8ba2-fab953e8e56a\" (UID: \"856f0eec-2077-45e2-8ba2-fab953e8e56a\") "
Nov 03 19:34:40.782255 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.782117    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/856f0eec-2077-45e2-8ba2-fab953e8e56a-webhook-cert\" (UniqueName: \"kubernetes.io/secret/856f0eec-2077-45e2-8ba2-fab953e8e56a-webhook-cert\") pod \"856f0eec-2077-45e2-8ba2-fab953e8e56a\" (UID: \"856f0eec-2077-45e2-8ba2-fab953e8e56a\") "
Nov 03 19:34:40.784324 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.784284    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856f0eec-2077-45e2-8ba2-fab953e8e56a-metrics-certs" pod "856f0eec-2077-45e2-8ba2-fab953e8e56a" (UID: "856f0eec-2077-45e2-8ba2-fab953e8e56a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:34:40.784324 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.784307    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856f0eec-2077-45e2-8ba2-fab953e8e56a-kube-api-access-b8gdh" pod "856f0eec-2077-45e2-8ba2-fab953e8e56a" (UID: "856f0eec-2077-45e2-8ba2-fab953e8e56a"). InnerVolumeSpecName "kube-api-access-b8gdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:34:40.784520 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.784314    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856f0eec-2077-45e2-8ba2-fab953e8e56a-webhook-cert" pod "856f0eec-2077-45e2-8ba2-fab953e8e56a" (UID: "856f0eec-2077-45e2-8ba2-fab953e8e56a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:34:40.805062 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.805031    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:34:40.883274 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.883203    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f835fba3-cf23-40c2-98d1-06435fd18080-kube-api-access-ngk2r\" (UniqueName: \"kubernetes.io/projected/f835fba3-cf23-40c2-98d1-06435fd18080-kube-api-access-ngk2r\") pod \"f835fba3-cf23-40c2-98d1-06435fd18080\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") "
Nov 03 19:34:40.883274 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.883236    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f835fba3-cf23-40c2-98d1-06435fd18080-istio-token\" (UniqueName: \"kubernetes.io/projected/f835fba3-cf23-40c2-98d1-06435fd18080-istio-token\") pod \"f835fba3-cf23-40c2-98d1-06435fd18080\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") "
Nov 03 19:34:40.883274 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.883269    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f835fba3-cf23-40c2-98d1-06435fd18080-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f835fba3-cf23-40c2-98d1-06435fd18080-local-certs\") pod \"f835fba3-cf23-40c2-98d1-06435fd18080\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") "
Nov 03 19:34:40.883537 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.883300    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f835fba3-cf23-40c2-98d1-06435fd18080-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f835fba3-cf23-40c2-98d1-06435fd18080-istio-csr-ca-configmap\") pod \"f835fba3-cf23-40c2-98d1-06435fd18080\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") "
Nov 03 19:34:40.883537 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.883343    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-istio-kubeconfig\") pod \"f835fba3-cf23-40c2-98d1-06435fd18080\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") "
Nov 03 19:34:40.883537 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.883378    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-istio-csr-dns-cert\") pod \"f835fba3-cf23-40c2-98d1-06435fd18080\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") "
Nov 03 19:34:40.883537 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.883410    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-cacerts\" (UniqueName: \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-cacerts\") pod \"f835fba3-cf23-40c2-98d1-06435fd18080\" (UID: \"f835fba3-cf23-40c2-98d1-06435fd18080\") "
Nov 03 19:34:40.883734 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.883547    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8gdh\" (UniqueName: \"kubernetes.io/projected/856f0eec-2077-45e2-8ba2-fab953e8e56a-kube-api-access-b8gdh\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:40.883734 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.883573    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/856f0eec-2077-45e2-8ba2-fab953e8e56a-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:40.883734 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.883582    5237 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/856f0eec-2077-45e2-8ba2-fab953e8e56a-webhook-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:40.884035 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.884001    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f835fba3-cf23-40c2-98d1-06435fd18080-istio-csr-ca-configmap" pod "f835fba3-cf23-40c2-98d1-06435fd18080" (UID: "f835fba3-cf23-40c2-98d1-06435fd18080"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:34:40.886000 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.885970    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f835fba3-cf23-40c2-98d1-06435fd18080-kube-api-access-ngk2r" pod "f835fba3-cf23-40c2-98d1-06435fd18080" (UID: "f835fba3-cf23-40c2-98d1-06435fd18080"). InnerVolumeSpecName "kube-api-access-ngk2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:34:40.886090 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.885970    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-cacerts" pod "f835fba3-cf23-40c2-98d1-06435fd18080" (UID: "f835fba3-cf23-40c2-98d1-06435fd18080"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:34:40.886090 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.886071    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f835fba3-cf23-40c2-98d1-06435fd18080-istio-token" pod "f835fba3-cf23-40c2-98d1-06435fd18080" (UID: "f835fba3-cf23-40c2-98d1-06435fd18080"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:34:40.886203 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.886179    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-istio-kubeconfig" pod "f835fba3-cf23-40c2-98d1-06435fd18080" (UID: "f835fba3-cf23-40c2-98d1-06435fd18080"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:34:40.886419 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.886402    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f835fba3-cf23-40c2-98d1-06435fd18080-local-certs" pod "f835fba3-cf23-40c2-98d1-06435fd18080" (UID: "f835fba3-cf23-40c2-98d1-06435fd18080"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:34:40.887651 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.887627    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-istio-csr-dns-cert" pod "f835fba3-cf23-40c2-98d1-06435fd18080" (UID: "f835fba3-cf23-40c2-98d1-06435fd18080"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:34:40.985397 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.984923    5237 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f835fba3-cf23-40c2-98d1-06435fd18080-istio-csr-ca-configmap\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:40.985397 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.984959    5237 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-istio-kubeconfig\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:40.985397 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.984974    5237 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-istio-csr-dns-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:40.985397 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.984989    5237 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f835fba3-cf23-40c2-98d1-06435fd18080-cacerts\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:40.985397 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.985006    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngk2r\" (UniqueName: \"kubernetes.io/projected/f835fba3-cf23-40c2-98d1-06435fd18080-kube-api-access-ngk2r\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:40.985397 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.985020    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f835fba3-cf23-40c2-98d1-06435fd18080-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:40.985397 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:40.985032    5237 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f835fba3-cf23-40c2-98d1-06435fd18080-local-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:41.433919 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.433887    5237 generic.go:356] "Generic (PLEG): container finished" podID="f835fba3-cf23-40c2-98d1-06435fd18080" containerID="081ad13c636149ecd1076bf7cf48a757728685b0aeb1fd80737110efce1ea543" exitCode=0
Nov 03 19:34:41.434372 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.433940    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"
Nov 03 19:34:41.434372 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.433962    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s" event={"ID":"f835fba3-cf23-40c2-98d1-06435fd18080","Type":"ContainerDied","Data":"081ad13c636149ecd1076bf7cf48a757728685b0aeb1fd80737110efce1ea543"}
Nov 03 19:34:41.434372 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.434015    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s" event={"ID":"f835fba3-cf23-40c2-98d1-06435fd18080","Type":"ContainerDied","Data":"80a8a1b8d79b05a857314cf5b44da47ecc5e3f3fc5842858033403267e6d8c99"}
Nov 03 19:34:41.434372 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.434040    5237 scope.go:122] "RemoveContainer" containerID="081ad13c636149ecd1076bf7cf48a757728685b0aeb1fd80737110efce1ea543"
Nov 03 19:34:41.436201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.435470    5237 generic.go:356] "Generic (PLEG): container finished" podID="856f0eec-2077-45e2-8ba2-fab953e8e56a" containerID="9a7d9cdfe48d9d8f12e20a6763c0c3fa02b413cd08185d5ff5a8ffd73af406c5" exitCode=0
Nov 03 19:34:41.436201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.436162    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-84p5p"
Nov 03 19:34:41.437034 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.436916    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-84p5p" event={"ID":"856f0eec-2077-45e2-8ba2-fab953e8e56a","Type":"ContainerDied","Data":"9a7d9cdfe48d9d8f12e20a6763c0c3fa02b413cd08185d5ff5a8ffd73af406c5"}
Nov 03 19:34:41.437034 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.436986    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-84p5p" event={"ID":"856f0eec-2077-45e2-8ba2-fab953e8e56a","Type":"ContainerDied","Data":"0dd174895f8331dcb015b761e1dd64ef7375d31f974da04ab08995187340b6e3"}
Nov 03 19:34:41.444452 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.444371    5237 scope.go:122] "RemoveContainer" containerID="081ad13c636149ecd1076bf7cf48a757728685b0aeb1fd80737110efce1ea543"
Nov 03 19:34:41.445050 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:34:41.444965    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081ad13c636149ecd1076bf7cf48a757728685b0aeb1fd80737110efce1ea543\": container with ID starting with 081ad13c636149ecd1076bf7cf48a757728685b0aeb1fd80737110efce1ea543 not found: ID does not exist" containerID="081ad13c636149ecd1076bf7cf48a757728685b0aeb1fd80737110efce1ea543"
Nov 03 19:34:41.445050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.445003    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081ad13c636149ecd1076bf7cf48a757728685b0aeb1fd80737110efce1ea543"} err="failed to get container status \"081ad13c636149ecd1076bf7cf48a757728685b0aeb1fd80737110efce1ea543\": rpc error: code = NotFound desc = could not find container \"081ad13c636149ecd1076bf7cf48a757728685b0aeb1fd80737110efce1ea543\": container with ID starting with 081ad13c636149ecd1076bf7cf48a757728685b0aeb1fd80737110efce1ea543 not found: ID does not exist"
Nov 03 19:34:41.445050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.445027    5237 scope.go:122] "RemoveContainer" containerID="9a7d9cdfe48d9d8f12e20a6763c0c3fa02b413cd08185d5ff5a8ffd73af406c5"
Nov 03 19:34:41.454144 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.454127    5237 scope.go:122] "RemoveContainer" containerID="9a7d9cdfe48d9d8f12e20a6763c0c3fa02b413cd08185d5ff5a8ffd73af406c5"
Nov 03 19:34:41.454439 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:34:41.454415    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a7d9cdfe48d9d8f12e20a6763c0c3fa02b413cd08185d5ff5a8ffd73af406c5\": container with ID starting with 9a7d9cdfe48d9d8f12e20a6763c0c3fa02b413cd08185d5ff5a8ffd73af406c5 not found: ID does not exist" containerID="9a7d9cdfe48d9d8f12e20a6763c0c3fa02b413cd08185d5ff5a8ffd73af406c5"
Nov 03 19:34:41.454505 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.454451    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7d9cdfe48d9d8f12e20a6763c0c3fa02b413cd08185d5ff5a8ffd73af406c5"} err="failed to get container status \"9a7d9cdfe48d9d8f12e20a6763c0c3fa02b413cd08185d5ff5a8ffd73af406c5\": rpc error: code = NotFound desc = could not find container \"9a7d9cdfe48d9d8f12e20a6763c0c3fa02b413cd08185d5ff5a8ffd73af406c5\": container with ID starting with 9a7d9cdfe48d9d8f12e20a6763c0c3fa02b413cd08185d5ff5a8ffd73af406c5 not found: ID does not exist"
Nov 03 19:34:41.460810 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.460790    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"]
Nov 03 19:34:41.467647 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.467626    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jq87s"]
Nov 03 19:34:41.484184 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.484157    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-84p5p"]
Nov 03 19:34:41.488816 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.488789    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-84p5p"]
Nov 03 19:34:41.858478 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.858434    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856f0eec-2077-45e2-8ba2-fab953e8e56a" path="/var/lib/kubelet/pods/856f0eec-2077-45e2-8ba2-fab953e8e56a/volumes"
Nov 03 19:34:41.858973 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:41.858949    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f835fba3-cf23-40c2-98d1-06435fd18080" path="/var/lib/kubelet/pods/f835fba3-cf23-40c2-98d1-06435fd18080/volumes"
Nov 03 19:34:43.260940 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:34:43.260896    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:34:46.461698 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:46.461662    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64" event={"ID":"f816ed7b-55b4-4f23-9fd7-28851ef00f1c","Type":"ContainerStarted","Data":"3bd44d2440de8c599da163e3b56ef87158111b7b1f03af6429c38f3457a46b5f"}
Nov 03 19:34:46.462093 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:46.461727    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"
Nov 03 19:34:46.486979 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:46.486920    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64" podStartSLOduration=1.9579285039999998 podStartE2EDuration="16.486904632s" totalImagesPullingTime="14.528976128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:34:46.48136511 +0000 UTC m=+442.735672337" watchObservedRunningTime="2026-11-03 19:34:46.486904632 +0000 UTC m=+442.741211857"
Nov 03 19:34:51.441882 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:51.441854    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-78kwl"
Nov 03 19:34:57.471278 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:34:57.471240    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4gb64"
Nov 03 19:35:00.681227 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:00.681183    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:00.816237 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:00.816196    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:01.929486 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:01.929449    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:02.007361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.007328    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"]
Nov 03 19:35:02.007640 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.007628    5237 cpu_manager.go:414] "containerMap: removing container" podUID="f835fba3-cf23-40c2-98d1-06435fd18080" containerName="discovery"
Nov 03 19:35:02.007696 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.007641    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f835fba3-cf23-40c2-98d1-06435fd18080" containerName="discovery"
Nov 03 19:35:02.007696 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.007652    5237 cpu_manager.go:414] "containerMap: removing container" podUID="856f0eec-2077-45e2-8ba2-fab953e8e56a" containerName="manager"
Nov 03 19:35:02.007696 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.007657    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856f0eec-2077-45e2-8ba2-fab953e8e56a" containerName="manager"
Nov 03 19:35:02.007799 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.007729    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="856f0eec-2077-45e2-8ba2-fab953e8e56a" containerName="manager"
Nov 03 19:35:02.007799 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.007738    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="f835fba3-cf23-40c2-98d1-06435fd18080" containerName="discovery"
Nov 03 19:35:02.013057 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.013031    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:02.023195 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.023139    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 19:35:02.023195 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.023174    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:35:02.023382 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.023214    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 19:35:02.023382 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.023298    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:35:02.025202 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.025183    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wm9nt\""
Nov 03 19:35:02.025928 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.025905    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"]
Nov 03 19:35:02.064065 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.064022    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:02.170509 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.170475    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvqrv\" (UniqueName: \"kubernetes.io/projected/83f05fa5-292c-42dc-8c0b-01babed2602f-kube-api-access-kvqrv\") pod \"maas-api-6d669c7fcd-2km2t\" (UID: \"83f05fa5-292c-42dc-8c0b-01babed2602f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:02.170509 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.170526    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83f05fa5-292c-42dc-8c0b-01babed2602f-maas-api-tls\") pod \"maas-api-6d669c7fcd-2km2t\" (UID: \"83f05fa5-292c-42dc-8c0b-01babed2602f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:02.170728 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.170544    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83f05fa5-292c-42dc-8c0b-01babed2602f-metrics-certs\") pod \"maas-api-6d669c7fcd-2km2t\" (UID: \"83f05fa5-292c-42dc-8c0b-01babed2602f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:02.170728 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.170676    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f05fa5-292c-42dc-8c0b-01babed2602f-openshift-service-ca\") pod \"maas-api-6d669c7fcd-2km2t\" (UID: \"83f05fa5-292c-42dc-8c0b-01babed2602f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:02.225688 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.225614    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:02.271722 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.271689    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvqrv\" (UniqueName: \"kubernetes.io/projected/83f05fa5-292c-42dc-8c0b-01babed2602f-kube-api-access-kvqrv\") pod \"maas-api-6d669c7fcd-2km2t\" (UID: \"83f05fa5-292c-42dc-8c0b-01babed2602f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:02.271897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.271729    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83f05fa5-292c-42dc-8c0b-01babed2602f-maas-api-tls\") pod \"maas-api-6d669c7fcd-2km2t\" (UID: \"83f05fa5-292c-42dc-8c0b-01babed2602f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:02.271897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.271754    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83f05fa5-292c-42dc-8c0b-01babed2602f-metrics-certs\") pod \"maas-api-6d669c7fcd-2km2t\" (UID: \"83f05fa5-292c-42dc-8c0b-01babed2602f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:02.271897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.271799    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f05fa5-292c-42dc-8c0b-01babed2602f-openshift-service-ca\") pod \"maas-api-6d669c7fcd-2km2t\" (UID: \"83f05fa5-292c-42dc-8c0b-01babed2602f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:02.272509 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.272487    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f05fa5-292c-42dc-8c0b-01babed2602f-openshift-service-ca\") pod \"maas-api-6d669c7fcd-2km2t\" (UID: \"83f05fa5-292c-42dc-8c0b-01babed2602f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:02.274234 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.274210    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83f05fa5-292c-42dc-8c0b-01babed2602f-maas-api-tls\") pod \"maas-api-6d669c7fcd-2km2t\" (UID: \"83f05fa5-292c-42dc-8c0b-01babed2602f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:02.274360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.274339    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83f05fa5-292c-42dc-8c0b-01babed2602f-metrics-certs\") pod \"maas-api-6d669c7fcd-2km2t\" (UID: \"83f05fa5-292c-42dc-8c0b-01babed2602f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:02.286013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.285984    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvqrv\" (UniqueName: \"kubernetes.io/projected/83f05fa5-292c-42dc-8c0b-01babed2602f-kube-api-access-kvqrv\") pod \"maas-api-6d669c7fcd-2km2t\" (UID: \"83f05fa5-292c-42dc-8c0b-01babed2602f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:02.328124 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.328086    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:02.612098 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:02.612061    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"]
Nov 03 19:35:02.615325 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:35:02.615297    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f05fa5_292c_42dc_8c0b_01babed2602f.slice/crio-6380197f3fc3b801790f0d7da7bfea3cef4e5ddbc6e68a025a09589440045ca9 WatchSource:0}: Error finding container 6380197f3fc3b801790f0d7da7bfea3cef4e5ddbc6e68a025a09589440045ca9: Status 404 returned error can't find the container with id 6380197f3fc3b801790f0d7da7bfea3cef4e5ddbc6e68a025a09589440045ca9
Nov 03 19:35:03.533480 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:03.533444    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t" event={"ID":"83f05fa5-292c-42dc-8c0b-01babed2602f","Type":"ContainerStarted","Data":"6380197f3fc3b801790f0d7da7bfea3cef4e5ddbc6e68a025a09589440045ca9"}
Nov 03 19:35:04.539021 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:04.538981    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t" event={"ID":"83f05fa5-292c-42dc-8c0b-01babed2602f","Type":"ContainerStarted","Data":"a8a8b439a9337664facb4a0f165dcea115866a6bf43d3db2f9aa71c472d2f52d"}
Nov 03 19:35:04.539402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:04.539131    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:04.564151 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:04.564100    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t" podStartSLOduration=1.826679257 podStartE2EDuration="3.564085707s" totalImagesPullingTime="1.73740645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:35:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:35:04.563039943 +0000 UTC m=+460.817347168" watchObservedRunningTime="2026-11-03 19:35:04.564085707 +0000 UTC m=+460.818392931"
Nov 03 19:35:10.548858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:10.548806    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:11.438706 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:11.438676    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"
Nov 03 19:35:12.169425 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:12.169391    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:12.312695 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:12.312655    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:13.255159 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:35:13.255129    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:35:21.682894 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:21.682861    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"]
Nov 03 19:35:21.683492 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:21.683069    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t" podUID="83f05fa5-292c-42dc-8c0b-01babed2602f" containerName="maas-api" containerID="cri-o://a8a8b439a9337664facb4a0f165dcea115866a6bf43d3db2f9aa71c472d2f52d" gracePeriod=30
Nov 03 19:35:21.922606 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:21.922580    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:21.939005 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:21.938934    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83f05fa5-292c-42dc-8c0b-01babed2602f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/83f05fa5-292c-42dc-8c0b-01babed2602f-metrics-certs\") pod \"83f05fa5-292c-42dc-8c0b-01babed2602f\" (UID: \"83f05fa5-292c-42dc-8c0b-01babed2602f\") "
Nov 03 19:35:21.939005 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:21.938983    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83f05fa5-292c-42dc-8c0b-01babed2602f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83f05fa5-292c-42dc-8c0b-01babed2602f-maas-api-tls\") pod \"83f05fa5-292c-42dc-8c0b-01babed2602f\" (UID: \"83f05fa5-292c-42dc-8c0b-01babed2602f\") "
Nov 03 19:35:21.941193 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:21.941166    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f05fa5-292c-42dc-8c0b-01babed2602f-metrics-certs" pod "83f05fa5-292c-42dc-8c0b-01babed2602f" (UID: "83f05fa5-292c-42dc-8c0b-01babed2602f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:35:21.941193 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:21.941185    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f05fa5-292c-42dc-8c0b-01babed2602f-maas-api-tls" pod "83f05fa5-292c-42dc-8c0b-01babed2602f" (UID: "83f05fa5-292c-42dc-8c0b-01babed2602f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:35:22.040108 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:22.040064    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f05fa5-292c-42dc-8c0b-01babed2602f-kube-api-access-kvqrv\" (UniqueName: \"kubernetes.io/projected/83f05fa5-292c-42dc-8c0b-01babed2602f-kube-api-access-kvqrv\") pod \"83f05fa5-292c-42dc-8c0b-01babed2602f\" (UID: \"83f05fa5-292c-42dc-8c0b-01babed2602f\") "
Nov 03 19:35:22.040313 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:22.040118    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83f05fa5-292c-42dc-8c0b-01babed2602f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f05fa5-292c-42dc-8c0b-01babed2602f-openshift-service-ca\") pod \"83f05fa5-292c-42dc-8c0b-01babed2602f\" (UID: \"83f05fa5-292c-42dc-8c0b-01babed2602f\") "
Nov 03 19:35:22.040388 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:22.040343    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83f05fa5-292c-42dc-8c0b-01babed2602f-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:35:22.040388 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:22.040363    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83f05fa5-292c-42dc-8c0b-01babed2602f-maas-api-tls\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:35:22.040485 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:22.040455    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f05fa5-292c-42dc-8c0b-01babed2602f-openshift-service-ca" pod "83f05fa5-292c-42dc-8c0b-01babed2602f" (UID: "83f05fa5-292c-42dc-8c0b-01babed2602f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:35:22.042173 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:22.042152    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f05fa5-292c-42dc-8c0b-01babed2602f-kube-api-access-kvqrv" pod "83f05fa5-292c-42dc-8c0b-01babed2602f" (UID: "83f05fa5-292c-42dc-8c0b-01babed2602f"). InnerVolumeSpecName "kube-api-access-kvqrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:35:22.140736 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:22.140695    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvqrv\" (UniqueName: \"kubernetes.io/projected/83f05fa5-292c-42dc-8c0b-01babed2602f-kube-api-access-kvqrv\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:35:22.140736 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:22.140730    5237 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f05fa5-292c-42dc-8c0b-01babed2602f-openshift-service-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:35:22.613762 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:22.613729    5237 generic.go:356] "Generic (PLEG): container finished" podID="83f05fa5-292c-42dc-8c0b-01babed2602f" containerID="a8a8b439a9337664facb4a0f165dcea115866a6bf43d3db2f9aa71c472d2f52d" exitCode=0
Nov 03 19:35:22.613957 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:22.613778    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"
Nov 03 19:35:22.613957 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:22.613812    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t" event={"ID":"83f05fa5-292c-42dc-8c0b-01babed2602f","Type":"ContainerDied","Data":"a8a8b439a9337664facb4a0f165dcea115866a6bf43d3db2f9aa71c472d2f52d"}
Nov 03 19:35:22.613957 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:22.613894    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t" event={"ID":"83f05fa5-292c-42dc-8c0b-01babed2602f","Type":"ContainerDied","Data":"6380197f3fc3b801790f0d7da7bfea3cef4e5ddbc6e68a025a09589440045ca9"}
Nov 03 19:35:22.613957 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:22.613917    5237 scope.go:122] "RemoveContainer" containerID="a8a8b439a9337664facb4a0f165dcea115866a6bf43d3db2f9aa71c472d2f52d"
Nov 03 19:35:22.622761 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:22.622746    5237 scope.go:122] "RemoveContainer" containerID="a8a8b439a9337664facb4a0f165dcea115866a6bf43d3db2f9aa71c472d2f52d"
Nov 03 19:35:22.623058 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:35:22.623038    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a8b439a9337664facb4a0f165dcea115866a6bf43d3db2f9aa71c472d2f52d\": container with ID starting with a8a8b439a9337664facb4a0f165dcea115866a6bf43d3db2f9aa71c472d2f52d not found: ID does not exist" containerID="a8a8b439a9337664facb4a0f165dcea115866a6bf43d3db2f9aa71c472d2f52d"
Nov 03 19:35:22.623156 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:22.623066    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a8b439a9337664facb4a0f165dcea115866a6bf43d3db2f9aa71c472d2f52d"} err="failed to get container status \"a8a8b439a9337664facb4a0f165dcea115866a6bf43d3db2f9aa71c472d2f52d\": rpc error: code = NotFound desc = could not find container \"a8a8b439a9337664facb4a0f165dcea115866a6bf43d3db2f9aa71c472d2f52d\": container with ID starting with a8a8b439a9337664facb4a0f165dcea115866a6bf43d3db2f9aa71c472d2f52d not found: ID does not exist"
Nov 03 19:35:22.635702 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:22.635677    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"]
Nov 03 19:35:22.639754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:22.639733    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2km2t"]
Nov 03 19:35:23.858632 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:23.858600    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83f05fa5-292c-42dc-8c0b-01babed2602f" path="/var/lib/kubelet/pods/83f05fa5-292c-42dc-8c0b-01babed2602f/volumes"
Nov 03 19:35:28.431513 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:28.431438    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:28.816438 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:28.816404    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:36.227932 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:36.227896    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-msxfh"]
Nov 03 19:35:36.228345 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:36.228232    5237 cpu_manager.go:414] "containerMap: removing container" podUID="83f05fa5-292c-42dc-8c0b-01babed2602f" containerName="maas-api"
Nov 03 19:35:36.228345 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:36.228244    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f05fa5-292c-42dc-8c0b-01babed2602f" containerName="maas-api"
Nov 03 19:35:36.228345 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:36.228306    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f05fa5-292c-42dc-8c0b-01babed2602f" containerName="maas-api"
Nov 03 19:35:36.232634 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:36.232617    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-msxfh"
Nov 03 19:35:36.240448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:36.240422    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-msxfh"]
Nov 03 19:35:36.337858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:36.337801    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pqs4\" (UniqueName: \"kubernetes.io/projected/8e2a098b-1b62-4639-bd21-bdf36b47db70-kube-api-access-9pqs4\") pod \"llmisvc-controller-manager-6466c554d8-msxfh\" (UID: \"8e2a098b-1b62-4639-bd21-bdf36b47db70\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-msxfh"
Nov 03 19:35:36.338037 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:36.337922    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e2a098b-1b62-4639-bd21-bdf36b47db70-cert\") pod \"llmisvc-controller-manager-6466c554d8-msxfh\" (UID: \"8e2a098b-1b62-4639-bd21-bdf36b47db70\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-msxfh"
Nov 03 19:35:36.438902 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:36.438869    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pqs4\" (UniqueName: \"kubernetes.io/projected/8e2a098b-1b62-4639-bd21-bdf36b47db70-kube-api-access-9pqs4\") pod \"llmisvc-controller-manager-6466c554d8-msxfh\" (UID: \"8e2a098b-1b62-4639-bd21-bdf36b47db70\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-msxfh"
Nov 03 19:35:36.439036 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:36.438928    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e2a098b-1b62-4639-bd21-bdf36b47db70-cert\") pod \"llmisvc-controller-manager-6466c554d8-msxfh\" (UID: \"8e2a098b-1b62-4639-bd21-bdf36b47db70\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-msxfh"
Nov 03 19:35:36.441276 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:36.441243    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e2a098b-1b62-4639-bd21-bdf36b47db70-cert\") pod \"llmisvc-controller-manager-6466c554d8-msxfh\" (UID: \"8e2a098b-1b62-4639-bd21-bdf36b47db70\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-msxfh"
Nov 03 19:35:36.452792 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:36.452762    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pqs4\" (UniqueName: \"kubernetes.io/projected/8e2a098b-1b62-4639-bd21-bdf36b47db70-kube-api-access-9pqs4\") pod \"llmisvc-controller-manager-6466c554d8-msxfh\" (UID: \"8e2a098b-1b62-4639-bd21-bdf36b47db70\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-msxfh"
Nov 03 19:35:36.547533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:36.547498    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-msxfh"
Nov 03 19:35:36.819998 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:36.819870    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-msxfh"]
Nov 03 19:35:36.823132 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:35:36.823100    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2a098b_1b62_4639_bd21_bdf36b47db70.slice/crio-d6ca94c8ae81cc305dbc5e7ebba5996b1d00f958cfd91db51d44d568cae31a75 WatchSource:0}: Error finding container d6ca94c8ae81cc305dbc5e7ebba5996b1d00f958cfd91db51d44d568cae31a75: Status 404 returned error can't find the container with id d6ca94c8ae81cc305dbc5e7ebba5996b1d00f958cfd91db51d44d568cae31a75
Nov 03 19:35:37.205792 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:37.205757    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:37.339080 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:37.339046    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:37.681561 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:37.681528    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-msxfh" event={"ID":"8e2a098b-1b62-4639-bd21-bdf36b47db70","Type":"ContainerStarted","Data":"fb6c5fecf1b639519894d55b2586941ba01b62ee16b6f308c4067d754177cc39"}
Nov 03 19:35:37.681745 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:37.681604    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-msxfh" event={"ID":"8e2a098b-1b62-4639-bd21-bdf36b47db70","Type":"ContainerStarted","Data":"d6ca94c8ae81cc305dbc5e7ebba5996b1d00f958cfd91db51d44d568cae31a75"}
Nov 03 19:35:37.681745 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:37.681642    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-msxfh"
Nov 03 19:35:37.700448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:35:37.700396    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-msxfh" podStartSLOduration=1.367701711 podStartE2EDuration="1.70038404s" totalImagesPullingTime="332.682329ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:35:37.699904687 +0000 UTC m=+493.954211913" watchObservedRunningTime="2026-11-03 19:35:37.70038404 +0000 UTC m=+493.954691264"
Nov 03 19:35:43.255119 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:35:43.255084    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:36:08.687761 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:08.687727    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-msxfh"
Nov 03 19:36:08.746630 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:08.746597    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"]
Nov 03 19:36:08.746831 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:08.746809    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk" podUID="d2e6c2a2-f9b1-4505-a970-d20a42adf795" containerName="manager" containerID="cri-o://69c77f11b793e2f6fe90de9b3f128ceade3d1afe61635ceed9494737a27233dd" gracePeriod=30
Nov 03 19:36:08.987388 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:08.987362    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"
Nov 03 19:36:09.116959 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:09.116927    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2e6c2a2-f9b1-4505-a970-d20a42adf795-kube-api-access-fvk5b\" (UniqueName: \"kubernetes.io/projected/d2e6c2a2-f9b1-4505-a970-d20a42adf795-kube-api-access-fvk5b\") pod \"d2e6c2a2-f9b1-4505-a970-d20a42adf795\" (UID: \"d2e6c2a2-f9b1-4505-a970-d20a42adf795\") "
Nov 03 19:36:09.117128 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:09.116985    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2e6c2a2-f9b1-4505-a970-d20a42adf795-cert\" (UniqueName: \"kubernetes.io/secret/d2e6c2a2-f9b1-4505-a970-d20a42adf795-cert\") pod \"d2e6c2a2-f9b1-4505-a970-d20a42adf795\" (UID: \"d2e6c2a2-f9b1-4505-a970-d20a42adf795\") "
Nov 03 19:36:09.118980 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:09.118948    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e6c2a2-f9b1-4505-a970-d20a42adf795-kube-api-access-fvk5b" pod "d2e6c2a2-f9b1-4505-a970-d20a42adf795" (UID: "d2e6c2a2-f9b1-4505-a970-d20a42adf795"). InnerVolumeSpecName "kube-api-access-fvk5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:36:09.119085 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:09.118986    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2e6c2a2-f9b1-4505-a970-d20a42adf795-cert" pod "d2e6c2a2-f9b1-4505-a970-d20a42adf795" (UID: "d2e6c2a2-f9b1-4505-a970-d20a42adf795"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:36:09.218089 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:09.218047    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvk5b\" (UniqueName: \"kubernetes.io/projected/d2e6c2a2-f9b1-4505-a970-d20a42adf795-kube-api-access-fvk5b\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:09.218089 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:09.218081    5237 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2e6c2a2-f9b1-4505-a970-d20a42adf795-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:09.621900 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:09.621863    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:09.805651 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:09.805613    5237 generic.go:356] "Generic (PLEG): container finished" podID="d2e6c2a2-f9b1-4505-a970-d20a42adf795" containerID="69c77f11b793e2f6fe90de9b3f128ceade3d1afe61635ceed9494737a27233dd" exitCode=0
Nov 03 19:36:09.806100 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:09.805660    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"
Nov 03 19:36:09.806100 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:09.805663    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk" event={"ID":"d2e6c2a2-f9b1-4505-a970-d20a42adf795","Type":"ContainerDied","Data":"69c77f11b793e2f6fe90de9b3f128ceade3d1afe61635ceed9494737a27233dd"}
Nov 03 19:36:09.806100 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:09.805715    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk" event={"ID":"d2e6c2a2-f9b1-4505-a970-d20a42adf795","Type":"ContainerDied","Data":"8a56e31ced81ab3e869eeccf122a2d7bb93bcaf69ea58476a65fd41b5ebbe1f1"}
Nov 03 19:36:09.806100 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:09.805744    5237 scope.go:122] "RemoveContainer" containerID="69c77f11b793e2f6fe90de9b3f128ceade3d1afe61635ceed9494737a27233dd"
Nov 03 19:36:09.814526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:09.814508    5237 scope.go:122] "RemoveContainer" containerID="69c77f11b793e2f6fe90de9b3f128ceade3d1afe61635ceed9494737a27233dd"
Nov 03 19:36:09.814794 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:36:09.814774    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c77f11b793e2f6fe90de9b3f128ceade3d1afe61635ceed9494737a27233dd\": container with ID starting with 69c77f11b793e2f6fe90de9b3f128ceade3d1afe61635ceed9494737a27233dd not found: ID does not exist" containerID="69c77f11b793e2f6fe90de9b3f128ceade3d1afe61635ceed9494737a27233dd"
Nov 03 19:36:09.814870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:09.814804    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c77f11b793e2f6fe90de9b3f128ceade3d1afe61635ceed9494737a27233dd"} err="failed to get container status \"69c77f11b793e2f6fe90de9b3f128ceade3d1afe61635ceed9494737a27233dd\": rpc error: code = NotFound desc = could not find container \"69c77f11b793e2f6fe90de9b3f128ceade3d1afe61635ceed9494737a27233dd\": container with ID starting with 69c77f11b793e2f6fe90de9b3f128ceade3d1afe61635ceed9494737a27233dd not found: ID does not exist"
Nov 03 19:36:09.828619 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:09.828596    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"]
Nov 03 19:36:09.835768 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:09.835747    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-h28gk"]
Nov 03 19:36:09.858018 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:09.857995    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2e6c2a2-f9b1-4505-a970-d20a42adf795" path="/var/lib/kubelet/pods/d2e6c2a2-f9b1-4505-a970-d20a42adf795/volumes"
Nov 03 19:36:11.375135 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:11.375104    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:13.255451 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:36:13.255348    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:36:27.663549 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:27.663506    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:28.889318 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:28.889283    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:29.745327 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:29.745279    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:30.259515 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:30.259471    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:31.047247 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:31.047197    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:32.598368 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:32.598320    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:35.458263 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:35.458173    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:36.993942 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:36.993903    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:37.802242 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:37.802207    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:39.190813 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:39.190779    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:43.255316 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:36:43.255283    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:36:54.294057 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:54.294019    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:54.434636 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:54.434597    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:55.496040 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.496005    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"]
Nov 03 19:36:55.496418 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.496324    5237 cpu_manager.go:414] "containerMap: removing container" podUID="d2e6c2a2-f9b1-4505-a970-d20a42adf795" containerName="manager"
Nov 03 19:36:55.496418 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.496335    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2e6c2a2-f9b1-4505-a970-d20a42adf795" containerName="manager"
Nov 03 19:36:55.496418 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.496405    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2e6c2a2-f9b1-4505-a970-d20a42adf795" containerName="manager"
Nov 03 19:36:55.498971 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.498945    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.512663 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.512638    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"]
Nov 03 19:36:55.587427 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.587394    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ndffd\""
Nov 03 19:36:55.589807 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.589783    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ec8c1c-49db-492e-958e-fabe09d044fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.589888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.589829    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ec8c1c-49db-492e-958e-fabe09d044fb-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.589938 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.589926    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ec8c1c-49db-492e-958e-fabe09d044fb-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.590016 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.589989    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ec8c1c-49db-492e-958e-fabe09d044fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.590078 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.590066    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ec8c1c-49db-492e-958e-fabe09d044fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.590125 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.590094    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ec8c1c-49db-492e-958e-fabe09d044fb-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.590125 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.590112    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgh4l\" (UniqueName: \"kubernetes.io/projected/98ec8c1c-49db-492e-958e-fabe09d044fb-kube-api-access-hgh4l\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.590224 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.590209    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ec8c1c-49db-492e-958e-fabe09d044fb-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.590263 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.590253    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ec8c1c-49db-492e-958e-fabe09d044fb-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.690775 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.690738    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ec8c1c-49db-492e-958e-fabe09d044fb-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.690775 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.690775    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgh4l\" (UniqueName: \"kubernetes.io/projected/98ec8c1c-49db-492e-958e-fabe09d044fb-kube-api-access-hgh4l\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.691053 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.690809    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ec8c1c-49db-492e-958e-fabe09d044fb-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.691053 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.690874    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ec8c1c-49db-492e-958e-fabe09d044fb-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.691053 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.690926    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ec8c1c-49db-492e-958e-fabe09d044fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.691053 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.690955    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ec8c1c-49db-492e-958e-fabe09d044fb-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.691053 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.690992    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ec8c1c-49db-492e-958e-fabe09d044fb-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.691302 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.691110    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ec8c1c-49db-492e-958e-fabe09d044fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.691302 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.691157    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ec8c1c-49db-492e-958e-fabe09d044fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.691302 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.691274    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ec8c1c-49db-492e-958e-fabe09d044fb-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.691461 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.691321    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ec8c1c-49db-492e-958e-fabe09d044fb-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.691461 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.691405    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ec8c1c-49db-492e-958e-fabe09d044fb-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.691594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.691463    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ec8c1c-49db-492e-958e-fabe09d044fb-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.691815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.691793    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ec8c1c-49db-492e-958e-fabe09d044fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.693484 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.693459    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ec8c1c-49db-492e-958e-fabe09d044fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.693590 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.693574    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ec8c1c-49db-492e-958e-fabe09d044fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.702980 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.702959    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ec8c1c-49db-492e-958e-fabe09d044fb-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.703076 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.703028    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgh4l\" (UniqueName: \"kubernetes.io/projected/98ec8c1c-49db-492e-958e-fabe09d044fb-kube-api-access-hgh4l\") pod \"maas-default-gateway-openshift-default-66db998887-f5xfb\" (UID: \"98ec8c1c-49db-492e-958e-fabe09d044fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:55.812446 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:55.812407    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:36:56.093632 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:56.093560    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"]
Nov 03 19:36:56.097207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:36:56.097179    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ec8c1c_49db_492e_958e_fabe09d044fb.slice/crio-99388a3346eb01f93610b7bd621e4342596205ff106400d6e054d6842cba70aa WatchSource:0}: Error finding container 99388a3346eb01f93610b7bd621e4342596205ff106400d6e054d6842cba70aa: Status 404 returned error can't find the container with id 99388a3346eb01f93610b7bd621e4342596205ff106400d6e054d6842cba70aa
Nov 03 19:36:56.995179 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:56.995144    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb" event={"ID":"98ec8c1c-49db-492e-958e-fabe09d044fb","Type":"ContainerStarted","Data":"99388a3346eb01f93610b7bd621e4342596205ff106400d6e054d6842cba70aa"}
Nov 03 19:36:59.926114 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:59.926083    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:36:59.926406 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:59.926153    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:36:59.926406 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:36:59.926186    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:37:00.011375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:00.011330    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb" event={"ID":"98ec8c1c-49db-492e-958e-fabe09d044fb","Type":"ContainerStarted","Data":"7dde7ce15a22753ffaba38482865f9a11f3c5b86e9a31c0dba6d36fd3eb87d6c"}
Nov 03 19:37:00.039383 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:00.039321    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb" podStartSLOduration=1.213161329 podStartE2EDuration="5.039302304s" totalImagesPullingTime="3.826140975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:00.03712386 +0000 UTC m=+576.291431088" watchObservedRunningTime="2026-11-03 19:37:00.039302304 +0000 UTC m=+576.293609530"
Nov 03 19:37:00.496748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:00.496697    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:00.654560 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:00.654528    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:00.813463 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:00.813429    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:37:00.814999 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:00.814966    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-f5xfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Nov 03 19:37:00.815127 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:00.815019    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb" podUID="98ec8c1c-49db-492e-958e-fabe09d044fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Nov 03 19:37:01.813193 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:01.813158    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-f5xfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Nov 03 19:37:01.813615 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:01.813225    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb" podUID="98ec8c1c-49db-492e-958e-fabe09d044fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Nov 03 19:37:02.813116 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:02.813079    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-f5xfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Nov 03 19:37:02.813293 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:02.813170    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb" podUID="98ec8c1c-49db-492e-958e-fabe09d044fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Nov 03 19:37:03.813389 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:03.813354    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-f5xfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Nov 03 19:37:03.813775 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:03.813401    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb" podUID="98ec8c1c-49db-492e-958e-fabe09d044fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Nov 03 19:37:04.813258 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:04.813223    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-f5xfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Nov 03 19:37:04.813412 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:04.813276    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb" podUID="98ec8c1c-49db-492e-958e-fabe09d044fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Nov 03 19:37:05.813789 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:05.813744    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-f5xfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Nov 03 19:37:05.814239 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:05.813803    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb" podUID="98ec8c1c-49db-492e-958e-fabe09d044fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Nov 03 19:37:06.813175 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:06.813136    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-f5xfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Nov 03 19:37:06.813389 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:06.813198    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb" podUID="98ec8c1c-49db-492e-958e-fabe09d044fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Nov 03 19:37:08.813803 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:08.813749    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-f5xfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:37:08.814341 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:08.813828    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb" podUID="98ec8c1c-49db-492e-958e-fabe09d044fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:37:09.856022 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:09.855984    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-f5xfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:37:09.856544 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:09.856040    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb" podUID="98ec8c1c-49db-492e-958e-fabe09d044fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:37:10.182067 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.181951    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:37:10.182272 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.182248    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:37:10.183348 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.183324    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb"
Nov 03 19:37:10.572827 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.572795    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"]
Nov 03 19:37:10.575988 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.575972    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.590419 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.590382    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"]
Nov 03 19:37:10.758153 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.758116    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d65667-e957-4334-a43a-71d03f1d1000-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.758153 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.758160    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d65667-e957-4334-a43a-71d03f1d1000-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.758361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.758179    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d65667-e957-4334-a43a-71d03f1d1000-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.758361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.758300    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d65667-e957-4334-a43a-71d03f1d1000-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.758361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.758344    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d65667-e957-4334-a43a-71d03f1d1000-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.758460 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.758362    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d65667-e957-4334-a43a-71d03f1d1000-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.758460 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.758409    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d65667-e957-4334-a43a-71d03f1d1000-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.758460 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.758435    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr9xt\" (UniqueName: \"kubernetes.io/projected/c8d65667-e957-4334-a43a-71d03f1d1000-kube-api-access-fr9xt\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.758552 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.758532    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d65667-e957-4334-a43a-71d03f1d1000-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.859078 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.858982    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d65667-e957-4334-a43a-71d03f1d1000-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.859078 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.859036    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d65667-e957-4334-a43a-71d03f1d1000-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.859078 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.859061    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d65667-e957-4334-a43a-71d03f1d1000-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.859633 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.859087    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d65667-e957-4334-a43a-71d03f1d1000-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.859633 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.859118    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr9xt\" (UniqueName: \"kubernetes.io/projected/c8d65667-e957-4334-a43a-71d03f1d1000-kube-api-access-fr9xt\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.859633 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.859167    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d65667-e957-4334-a43a-71d03f1d1000-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.859633 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.859202    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d65667-e957-4334-a43a-71d03f1d1000-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.859633 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.859246    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d65667-e957-4334-a43a-71d03f1d1000-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.859633 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.859477    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d65667-e957-4334-a43a-71d03f1d1000-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.859633 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.859570    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d65667-e957-4334-a43a-71d03f1d1000-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.859633 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.859598    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d65667-e957-4334-a43a-71d03f1d1000-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.859972 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.859688    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d65667-e957-4334-a43a-71d03f1d1000-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.859972 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.859871    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d65667-e957-4334-a43a-71d03f1d1000-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.859972 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.859940    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d65667-e957-4334-a43a-71d03f1d1000-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.861568 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.861544    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d65667-e957-4334-a43a-71d03f1d1000-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.861797 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.861776    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d65667-e957-4334-a43a-71d03f1d1000-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.872025 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.872006    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr9xt\" (UniqueName: \"kubernetes.io/projected/c8d65667-e957-4334-a43a-71d03f1d1000-kube-api-access-fr9xt\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.872108 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.872053    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d65667-e957-4334-a43a-71d03f1d1000-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-t9ft8\" (UID: \"c8d65667-e957-4334-a43a-71d03f1d1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:10.889889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:10.889869    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:11.172256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:11.172230    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"]
Nov 03 19:37:11.174822 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:37:11.174795    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d65667_e957_4334_a43a_71d03f1d1000.slice/crio-d66861081edc06b6db870342d0154fc9b86f3e76697aea2987ee9cb108d3981a WatchSource:0}: Error finding container d66861081edc06b6db870342d0154fc9b86f3e76697aea2987ee9cb108d3981a: Status 404 returned error can't find the container with id d66861081edc06b6db870342d0154fc9b86f3e76697aea2987ee9cb108d3981a
Nov 03 19:37:11.177915 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:11.177880    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:37:11.178028 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:11.177961    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:37:11.178028 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:11.178004    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:37:12.061525 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:12.061486    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8" event={"ID":"c8d65667-e957-4334-a43a-71d03f1d1000","Type":"ContainerStarted","Data":"6838b666820d95e68aff251a5f8393c6d941c7e72bb690288075ff5d9da95418"}
Nov 03 19:37:12.062011 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:12.061543    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8" event={"ID":"c8d65667-e957-4334-a43a-71d03f1d1000","Type":"ContainerStarted","Data":"d66861081edc06b6db870342d0154fc9b86f3e76697aea2987ee9cb108d3981a"}
Nov 03 19:37:12.123323 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:12.123267    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8" podStartSLOduration=2.123253785 podStartE2EDuration="2.123253785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:12.121545812 +0000 UTC m=+588.375853037" watchObservedRunningTime="2026-11-03 19:37:12.123253785 +0000 UTC m=+588.377561009"
Nov 03 19:37:12.893067 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:12.893024    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:12.894546 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:12.894520    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-t9ft8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Nov 03 19:37:12.894685 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:12.894577    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8" podUID="c8d65667-e957-4334-a43a-71d03f1d1000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Nov 03 19:37:13.254958 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:37:13.254875    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:37:13.893259 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:13.893222    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-t9ft8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Nov 03 19:37:13.893438 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:13.893279    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8" podUID="c8d65667-e957-4334-a43a-71d03f1d1000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Nov 03 19:37:14.893319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:14.893275    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-t9ft8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Nov 03 19:37:14.893796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:14.893339    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8" podUID="c8d65667-e957-4334-a43a-71d03f1d1000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Nov 03 19:37:15.891084 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:15.891044    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-t9ft8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Nov 03 19:37:15.891417 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:15.891385    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8" podUID="c8d65667-e957-4334-a43a-71d03f1d1000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Nov 03 19:37:16.891285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:16.891248    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-t9ft8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Nov 03 19:37:16.891964 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:16.891931    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8" podUID="c8d65667-e957-4334-a43a-71d03f1d1000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Nov 03 19:37:17.377113 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:17.377078    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:17.515856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:17.515806    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:17.890650 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:17.890606    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-t9ft8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Nov 03 19:37:17.890885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:17.890659    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8" podUID="c8d65667-e957-4334-a43a-71d03f1d1000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Nov 03 19:37:19.300392 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:19.300358    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:19.301055 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:19.301035    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:19.301711 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:19.301691    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8"
Nov 03 19:37:25.602949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.602916    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"]
Nov 03 19:37:25.610830 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.610809    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.621223 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.621192    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"]
Nov 03 19:37:25.695157 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.695126    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5da6be6-a4c0-40f5-8f07-d3de1d188394-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.695157 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.695163    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5da6be6-a4c0-40f5-8f07-d3de1d188394-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.695378 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.695182    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5da6be6-a4c0-40f5-8f07-d3de1d188394-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.695378 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.695263    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7b7\" (UniqueName: \"kubernetes.io/projected/f5da6be6-a4c0-40f5-8f07-d3de1d188394-kube-api-access-st7b7\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.695378 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.695323    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5da6be6-a4c0-40f5-8f07-d3de1d188394-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.695506 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.695379    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5da6be6-a4c0-40f5-8f07-d3de1d188394-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.695506 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.695407    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5da6be6-a4c0-40f5-8f07-d3de1d188394-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.695506 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.695448    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5da6be6-a4c0-40f5-8f07-d3de1d188394-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.695802 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.695575    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5da6be6-a4c0-40f5-8f07-d3de1d188394-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.796671 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.796632    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7b7\" (UniqueName: \"kubernetes.io/projected/f5da6be6-a4c0-40f5-8f07-d3de1d188394-kube-api-access-st7b7\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.796913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.796682    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5da6be6-a4c0-40f5-8f07-d3de1d188394-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.796913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.796710    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5da6be6-a4c0-40f5-8f07-d3de1d188394-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.796913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.796747    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5da6be6-a4c0-40f5-8f07-d3de1d188394-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.796913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.796803    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5da6be6-a4c0-40f5-8f07-d3de1d188394-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.796913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.796857    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5da6be6-a4c0-40f5-8f07-d3de1d188394-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.796913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.796913    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5da6be6-a4c0-40f5-8f07-d3de1d188394-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.797225 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.796949    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5da6be6-a4c0-40f5-8f07-d3de1d188394-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.797225 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.796978    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5da6be6-a4c0-40f5-8f07-d3de1d188394-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.797225 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.797175    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5da6be6-a4c0-40f5-8f07-d3de1d188394-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.797370 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.797288    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5da6be6-a4c0-40f5-8f07-d3de1d188394-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.797437 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.797417    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5da6be6-a4c0-40f5-8f07-d3de1d188394-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.797478 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.797461    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5da6be6-a4c0-40f5-8f07-d3de1d188394-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.797570 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.797552    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5da6be6-a4c0-40f5-8f07-d3de1d188394-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.799272 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.799256    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5da6be6-a4c0-40f5-8f07-d3de1d188394-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.799479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.799460    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5da6be6-a4c0-40f5-8f07-d3de1d188394-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.809424 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.809396    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5da6be6-a4c0-40f5-8f07-d3de1d188394-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.809654 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.809636    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7b7\" (UniqueName: \"kubernetes.io/projected/f5da6be6-a4c0-40f5-8f07-d3de1d188394-kube-api-access-st7b7\") pod \"maas-default-gateway-openshift-default-66db998887-bqzwl\" (UID: \"f5da6be6-a4c0-40f5-8f07-d3de1d188394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:25.926470 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:25.926390    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:26.205163 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:26.205132    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"]
Nov 03 19:37:26.207750 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:37:26.207724    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5da6be6_a4c0_40f5_8f07_d3de1d188394.slice/crio-c5a386d1e68ea1c466f4bd6f4b08fb2cec5a9ce3441a80f4b7a5946c95c399a1 WatchSource:0}: Error finding container c5a386d1e68ea1c466f4bd6f4b08fb2cec5a9ce3441a80f4b7a5946c95c399a1: Status 404 returned error can't find the container with id c5a386d1e68ea1c466f4bd6f4b08fb2cec5a9ce3441a80f4b7a5946c95c399a1
Nov 03 19:37:26.209816 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:26.209784    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:37:26.209926 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:26.209888    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:37:26.209926 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:26.209916    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:37:27.032200 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:27.032168    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:27.146761 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:27.146722    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl" event={"ID":"f5da6be6-a4c0-40f5-8f07-d3de1d188394","Type":"ContainerStarted","Data":"6b4068710d053ea102c04ecb7e3ba0f5f00779f9191db3b994b5f357b8c0a061"}
Nov 03 19:37:27.146980 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:27.146780    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl" event={"ID":"f5da6be6-a4c0-40f5-8f07-d3de1d188394","Type":"ContainerStarted","Data":"c5a386d1e68ea1c466f4bd6f4b08fb2cec5a9ce3441a80f4b7a5946c95c399a1"}
Nov 03 19:37:27.191394 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:27.191337    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl" podStartSLOduration=2.191317139 podStartE2EDuration="2.191317139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:27.189818851 +0000 UTC m=+603.444126081" watchObservedRunningTime="2026-11-03 19:37:27.191317139 +0000 UTC m=+603.445624372"
Nov 03 19:37:27.543354 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:27.543314    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:27.927497 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:27.927417    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:27.928852 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:27.928815    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-bqzwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 19:37:27.928971 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:27.928884    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl" podUID="f5da6be6-a4c0-40f5-8f07-d3de1d188394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 19:37:28.926911 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:28.926874    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-bqzwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 19:37:28.927325 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:28.926927    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl" podUID="f5da6be6-a4c0-40f5-8f07-d3de1d188394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 19:37:28.936790 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:28.936750    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:29.181625 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:29.181515    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:29.342527 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:29.342494    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:29.927172 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:29.927142    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-bqzwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 19:37:29.927641 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:29.927613    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl" podUID="f5da6be6-a4c0-40f5-8f07-d3de1d188394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 19:37:30.816896 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:30.816851    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:30.927071 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:30.927037    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-bqzwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 19:37:30.927236 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:30.927094    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl" podUID="f5da6be6-a4c0-40f5-8f07-d3de1d188394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 19:37:30.951346 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:30.951295    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:31.927559 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:31.927517    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-bqzwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 19:37:31.928009 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:31.927575    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl" podUID="f5da6be6-a4c0-40f5-8f07-d3de1d188394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 19:37:32.927679 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:32.927642    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-bqzwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 19:37:32.928095 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:32.927700    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl" podUID="f5da6be6-a4c0-40f5-8f07-d3de1d188394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 19:37:34.364317 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:34.364288    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:34.364803 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:34.364668    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:34.365536 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:34.365519    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl"
Nov 03 19:37:43.253788 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:37:43.253735    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:37:53.712474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:53.712437    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:53.844734 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:53.844700    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:54.326216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:54.326178    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:54.643884 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:54.643790    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:54.802690 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:54.802653    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:54.971549 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:54.971204    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:55.123934 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:55.123898    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:55.251214 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:55.251124    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:58.119780 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:58.119731    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:58.261114 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:58.261083    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:58.509938 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:58.509859    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:58.735124 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:58.735088    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:58.912164 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:58.912127    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:59.061222 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:59.061190    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:59.462795 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:59.462758    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:59.611169 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:37:59.610965    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:00.147444 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.147408    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"]
Nov 03 19:38:00.150936 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.150913    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:38:00.162516 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.162493    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:38:00.166079 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.166061    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:38:00.167867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.167820    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-036184-serving-cert\""
Nov 03 19:38:00.167981 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.167894    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wm9nt\""
Nov 03 19:38:00.167981 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.167961    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 19:38:00.169793 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.169766    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"]
Nov 03 19:38:00.288193 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.288159    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:00.315011 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.314977    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f48a7944-1a16-42d6-8483-e6d2533ca2b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-036184-645546498b-ml69b\" (UID: \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:38:00.315011 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.315015    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f48a7944-1a16-42d6-8483-e6d2533ca2b1-metrics-certs\") pod \"maas-api-e2e-worker-w3-036184-645546498b-ml69b\" (UID: \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:38:00.315287 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.315035    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f48a7944-1a16-42d6-8483-e6d2533ca2b1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-036184-645546498b-ml69b\" (UID: \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:38:00.315287 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.315189    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44ncq\" (UniqueName: \"kubernetes.io/projected/f48a7944-1a16-42d6-8483-e6d2533ca2b1-kube-api-access-44ncq\") pod \"maas-api-e2e-worker-w3-036184-645546498b-ml69b\" (UID: \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:38:00.422645 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.420120    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f48a7944-1a16-42d6-8483-e6d2533ca2b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-036184-645546498b-ml69b\" (UID: \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:38:00.422645 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.420179    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f48a7944-1a16-42d6-8483-e6d2533ca2b1-metrics-certs\") pod \"maas-api-e2e-worker-w3-036184-645546498b-ml69b\" (UID: \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:38:00.422645 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.420207    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f48a7944-1a16-42d6-8483-e6d2533ca2b1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-036184-645546498b-ml69b\" (UID: \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:38:00.422645 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.420289    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44ncq\" (UniqueName: \"kubernetes.io/projected/f48a7944-1a16-42d6-8483-e6d2533ca2b1-kube-api-access-44ncq\") pod \"maas-api-e2e-worker-w3-036184-645546498b-ml69b\" (UID: \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:38:00.422645 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.421604    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f48a7944-1a16-42d6-8483-e6d2533ca2b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-036184-645546498b-ml69b\" (UID: \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:38:00.437783 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.437743    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f48a7944-1a16-42d6-8483-e6d2533ca2b1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-036184-645546498b-ml69b\" (UID: \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:38:00.445500 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.445465    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f48a7944-1a16-42d6-8483-e6d2533ca2b1-metrics-certs\") pod \"maas-api-e2e-worker-w3-036184-645546498b-ml69b\" (UID: \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:38:00.458357 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.455785    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44ncq\" (UniqueName: \"kubernetes.io/projected/f48a7944-1a16-42d6-8483-e6d2533ca2b1-kube-api-access-44ncq\") pod \"maas-api-e2e-worker-w3-036184-645546498b-ml69b\" (UID: \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:38:00.464654 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.464245    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:38:00.490872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.490827    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:00.765950 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.765875    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"]
Nov 03 19:38:00.769010 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:38:00.768979    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48a7944_1a16_42d6_8483_e6d2533ca2b1.slice/crio-27a90eb63e37b1dd713655b8e9c04a117f0c1afb484d8bcf2a9e51e3e8ea1a8f WatchSource:0}: Error finding container 27a90eb63e37b1dd713655b8e9c04a117f0c1afb484d8bcf2a9e51e3e8ea1a8f: Status 404 returned error can't find the container with id 27a90eb63e37b1dd713655b8e9c04a117f0c1afb484d8bcf2a9e51e3e8ea1a8f
Nov 03 19:38:00.809005 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.808948    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:00.959933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:00.959892    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:01.296177 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.296135    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b" event={"ID":"f48a7944-1a16-42d6-8483-e6d2533ca2b1","Type":"ContainerStarted","Data":"a6e0a12cf8d1de9d6c5156f87dd314a10ae094c7f3cda6cbb684fd66824df675"}
Nov 03 19:38:01.296381 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.296247    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:38:01.296381 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.296273    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b" event={"ID":"f48a7944-1a16-42d6-8483-e6d2533ca2b1","Type":"ContainerStarted","Data":"27a90eb63e37b1dd713655b8e9c04a117f0c1afb484d8bcf2a9e51e3e8ea1a8f"}
Nov 03 19:38:01.320144 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.320086    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b" podStartSLOduration=0.915432087 podStartE2EDuration="1.320072851s" totalImagesPullingTime="404.640764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:01.319382468 +0000 UTC m=+637.573689692" watchObservedRunningTime="2026-11-03 19:38:01.320072851 +0000 UTC m=+637.574380080"
Nov 03 19:38:01.478198 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.478103    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"]
Nov 03 19:38:01.482650 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.482620    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"]
Nov 03 19:38:01.482898 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.482737    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:38:01.487019 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.486995    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-ef8f84-serving-cert\""
Nov 03 19:38:01.565927 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.565880    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:01.636139 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.636099    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bde0d49-ee70-4f96-991b-2f38e8984dcf-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh\" (UID: \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:38:01.636139 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.636141    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bde0d49-ee70-4f96-991b-2f38e8984dcf-metrics-certs\") pod \"maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh\" (UID: \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:38:01.636393 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.636229    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bde0d49-ee70-4f96-991b-2f38e8984dcf-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh\" (UID: \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:38:01.636393 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.636276    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmmnn\" (UniqueName: \"kubernetes.io/projected/7bde0d49-ee70-4f96-991b-2f38e8984dcf-kube-api-access-zmmnn\") pod \"maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh\" (UID: \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:38:01.721380 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.721341    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:01.737097 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.737026    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bde0d49-ee70-4f96-991b-2f38e8984dcf-metrics-certs\") pod \"maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh\" (UID: \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:38:01.737097 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.737073    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bde0d49-ee70-4f96-991b-2f38e8984dcf-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh\" (UID: \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:38:01.737289 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.737117    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmmnn\" (UniqueName: \"kubernetes.io/projected/7bde0d49-ee70-4f96-991b-2f38e8984dcf-kube-api-access-zmmnn\") pod \"maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh\" (UID: \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:38:01.737289 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.737216    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bde0d49-ee70-4f96-991b-2f38e8984dcf-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh\" (UID: \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:38:01.737855 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.737805    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bde0d49-ee70-4f96-991b-2f38e8984dcf-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh\" (UID: \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:38:01.739638 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.739615    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bde0d49-ee70-4f96-991b-2f38e8984dcf-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh\" (UID: \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:38:01.739739 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.739656    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bde0d49-ee70-4f96-991b-2f38e8984dcf-metrics-certs\") pod \"maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh\" (UID: \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:38:01.749716 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.749680    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmmnn\" (UniqueName: \"kubernetes.io/projected/7bde0d49-ee70-4f96-991b-2f38e8984dcf-kube-api-access-zmmnn\") pod \"maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh\" (UID: \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:38:01.799179 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:01.797448    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:38:02.089356 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:02.089328    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"]
Nov 03 19:38:02.092022 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:38:02.091989    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bde0d49_ee70_4f96_991b_2f38e8984dcf.slice/crio-c2fcfba0e8b61580d0c13f15b05095aa863715f12fe01bbdb9270aa04f4cc6a8 WatchSource:0}: Error finding container c2fcfba0e8b61580d0c13f15b05095aa863715f12fe01bbdb9270aa04f4cc6a8: Status 404 returned error can't find the container with id c2fcfba0e8b61580d0c13f15b05095aa863715f12fe01bbdb9270aa04f4cc6a8
Nov 03 19:38:02.301621 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:02.301580    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh" event={"ID":"7bde0d49-ee70-4f96-991b-2f38e8984dcf","Type":"ContainerStarted","Data":"c2fcfba0e8b61580d0c13f15b05095aa863715f12fe01bbdb9270aa04f4cc6a8"}
Nov 03 19:38:02.378347 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:02.378231    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:02.562916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:02.562885    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:02.696420 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:02.696387    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:02.868340 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:02.868307    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:03.011425 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.011326    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:03.310285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.310252    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh" event={"ID":"7bde0d49-ee70-4f96-991b-2f38e8984dcf","Type":"ContainerStarted","Data":"c890ce84d4dbbc2ce66343e08082f2e5584e14274716901550f3392533e647d9"}
Nov 03 19:38:03.310461 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.310390    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:38:03.331906 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.331851    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh" podStartSLOduration=1.916320331 podStartE2EDuration="2.331818294s" totalImagesPullingTime="415.497963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:03.330425285 +0000 UTC m=+639.584732510" watchObservedRunningTime="2026-11-03 19:38:03.331818294 +0000 UTC m=+639.586125519"
Nov 03 19:38:03.551952 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.551922    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"]
Nov 03 19:38:03.555373 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.555352    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:38:03.559476 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.559452    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-435bd8-serving-cert\""
Nov 03 19:38:03.569568 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.569543    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"]
Nov 03 19:38:03.596156 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.596120    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:03.673054 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.673023    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29vs7\" (UniqueName: \"kubernetes.io/projected/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-kube-api-access-29vs7\") pod \"maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68\" (UID: \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:38:03.673222 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.673129    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-metrics-certs\") pod \"maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68\" (UID: \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:38:03.673222 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.673180    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-maas-api-tls\") pod \"maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68\" (UID: \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:38:03.673222 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.673205    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68\" (UID: \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:38:03.754195 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.754047    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:03.774870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.774793    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-maas-api-tls\") pod \"maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68\" (UID: \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:38:03.775037 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.774898    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68\" (UID: \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:38:03.775037 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.774973    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29vs7\" (UniqueName: \"kubernetes.io/projected/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-kube-api-access-29vs7\") pod \"maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68\" (UID: \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:38:03.775037 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.775022    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-metrics-certs\") pod \"maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68\" (UID: \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:38:03.778088 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.778056    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68\" (UID: \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:38:03.778719 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.778691    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-metrics-certs\") pod \"maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68\" (UID: \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:38:03.778960 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.778941    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-maas-api-tls\") pod \"maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68\" (UID: \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:38:03.789337 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.789304    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29vs7\" (UniqueName: \"kubernetes.io/projected/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-kube-api-access-29vs7\") pod \"maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68\" (UID: \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:38:03.871962 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:03.871873    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:38:04.149228 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.149013    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"]
Nov 03 19:38:04.151655 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:38:04.151626    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbc3df5_49ef_4bbf_8a95_726f0d42f5ae.slice/crio-2324e79e0ffbf7167a182df3c9d58595d9b1a93ccc9f4385deab930f141d0630 WatchSource:0}: Error finding container 2324e79e0ffbf7167a182df3c9d58595d9b1a93ccc9f4385deab930f141d0630: Status 404 returned error can't find the container with id 2324e79e0ffbf7167a182df3c9d58595d9b1a93ccc9f4385deab930f141d0630
Nov 03 19:38:04.315564 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.315531    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68" event={"ID":"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae","Type":"ContainerStarted","Data":"2324e79e0ffbf7167a182df3c9d58595d9b1a93ccc9f4385deab930f141d0630"}
Nov 03 19:38:04.455365 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.455281    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"]
Nov 03 19:38:04.460591 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.460563    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:38:04.471222 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.471196    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-d8a4ea-serving-cert\""
Nov 03 19:38:04.471696 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.471580    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"]
Nov 03 19:38:04.546338 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.546302    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:04.597859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.591421    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-metrics-certs\") pod \"maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f\" (UID: \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:38:04.597859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.591496    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f\" (UID: \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:38:04.597859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.591534    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f\" (UID: \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:38:04.597859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.591571    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2489t\" (UniqueName: \"kubernetes.io/projected/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-kube-api-access-2489t\") pod \"maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f\" (UID: \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:38:04.684809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.684778    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:04.692871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.692846    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f\" (UID: \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:38:04.692952 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.692886    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f\" (UID: \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:38:04.692952 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.692911    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2489t\" (UniqueName: \"kubernetes.io/projected/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-kube-api-access-2489t\") pod \"maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f\" (UID: \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:38:04.693038 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.692983    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-metrics-certs\") pod \"maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f\" (UID: \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:38:04.693086 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:38:04.693045    5237 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-serving-cert: secret "maas-api-e2e-worker-w4-d8a4ea-serving-cert" not found
Nov 03 19:38:04.693147 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:38:04.693121    5237 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-maas-api-tls podName:ac52ab6c-a555-494b-b6b9-654d2d60c8e8 nodeName:}" failed. No retries permitted until 2026-11-03 19:38:05.193099111 +0000 UTC m=+641.447406328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-maas-api-tls") pod "maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f" (UID: "ac52ab6c-a555-494b-b6b9-654d2d60c8e8") : secret "maas-api-e2e-worker-w4-d8a4ea-serving-cert" not found
Nov 03 19:38:04.693399 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.693383    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f\" (UID: \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:38:04.695379 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.695361    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-metrics-certs\") pod \"maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f\" (UID: \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:38:04.709950 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:04.709869    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2489t\" (UniqueName: \"kubernetes.io/projected/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-kube-api-access-2489t\") pod \"maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f\" (UID: \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:38:05.042966 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:05.042932    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:05.198342 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:05.198306    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f\" (UID: \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:38:05.200729 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:05.200706    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f\" (UID: \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:38:05.320877 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:05.320785    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68" event={"ID":"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae","Type":"ContainerStarted","Data":"825a6dd97ad1b2a6cb1b31a1267c4636ac4a7bba986165f4474570890b06976b"}
Nov 03 19:38:05.321035 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:05.320902    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:38:05.346068 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:05.346013    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68" podStartSLOduration=1.9636640079999998 podStartE2EDuration="2.345998485s" totalImagesPullingTime="382.334477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:05.34367422 +0000 UTC m=+641.597981445" watchObservedRunningTime="2026-11-03 19:38:05.345998485 +0000 UTC m=+641.600305709"
Nov 03 19:38:05.379857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:05.377025    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:38:05.566637 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:05.566601    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:05.688059 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:05.687968    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"]
Nov 03 19:38:05.692900 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:38:05.692866    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac52ab6c_a555_494b_b6b9_654d2d60c8e8.slice/crio-3595e531fc8cfe65a02dedd54cce8cc843d33552312c29defc37a23a3be0a172 WatchSource:0}: Error finding container 3595e531fc8cfe65a02dedd54cce8cc843d33552312c29defc37a23a3be0a172: Status 404 returned error can't find the container with id 3595e531fc8cfe65a02dedd54cce8cc843d33552312c29defc37a23a3be0a172
Nov 03 19:38:05.695202 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:05.695181    5237 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:38:05.703413 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:05.703386    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:06.327634 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:06.327596    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f" event={"ID":"ac52ab6c-a555-494b-b6b9-654d2d60c8e8","Type":"ContainerStarted","Data":"a67dd756a8ba0b77313a9f5ef547801fd397e51b00ce17f326436f3949ddbf7e"}
Nov 03 19:38:06.327811 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:06.327656    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f" event={"ID":"ac52ab6c-a555-494b-b6b9-654d2d60c8e8","Type":"ContainerStarted","Data":"3595e531fc8cfe65a02dedd54cce8cc843d33552312c29defc37a23a3be0a172"}
Nov 03 19:38:06.327896 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:06.327870    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:38:06.352418 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:06.352364    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f" podStartSLOduration=1.9227575959999998 podStartE2EDuration="2.35234733s" totalImagesPullingTime="429.589734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:06.35037809 +0000 UTC m=+642.604685314" watchObservedRunningTime="2026-11-03 19:38:06.35234733 +0000 UTC m=+642.606654555"
Nov 03 19:38:06.352722 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:06.352695    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:06.478857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:06.478739    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:06.612305 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:06.612264    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:07.129753 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:07.129709    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:07.309704 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:07.309679    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:38:07.822201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:07.822164    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:08.794650 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:08.794611    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:09.321617 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:09.321591    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:38:09.445666 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:09.445622    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:10.323700 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:10.323665    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:11.123998 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:11.123963    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:11.333674 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:11.333648    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:38:11.822390 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:11.822353    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:12.339417 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:12.339392    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:38:12.438005 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:12.437973    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:13.031474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:13.031443    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:13.167157 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:13.167121    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:13.256020 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:38:13.255987    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:38:13.645012 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:13.644980    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:13.798602 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:13.798570    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:13.941878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:13.941753    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:14.456130 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:14.453919    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:15.381300 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.381249    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:15.558782 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.558751    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"]
Nov 03 19:38:15.562198 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.562171    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.568303 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.568281    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-036184\"/\"openshift-service-ca.crt\""
Nov 03 19:38:15.571035 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.571013    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-036184\"/\"kube-root-ca.crt\""
Nov 03 19:38:15.572677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.572655    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-036184\"/\"default-dockercfg-zqxbt\""
Nov 03 19:38:15.572794 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.572756    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-036184\"/\"simulator-w3-036184-kserve-self-signed-certs\""
Nov 03 19:38:15.573395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.573372    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"]
Nov 03 19:38:15.618379 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.618339    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-tmp-dir\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.618379 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.618375    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-dshm\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.618633 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.618414    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfvj8\" (UniqueName: \"kubernetes.io/projected/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-kube-api-access-lfvj8\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.618633 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.618488    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-model-cache\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.618633 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.618592    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-home\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.618799 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.618683    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-tls-certs\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.633051 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.632951    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:15.720166 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.720132    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-tls-certs\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.720166 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.720170    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-tmp-dir\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.720418 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.720188    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-dshm\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.720418 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.720223    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfvj8\" (UniqueName: \"kubernetes.io/projected/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-kube-api-access-lfvj8\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.720418 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.720241    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-model-cache\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.720418 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.720261    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-home\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.720735 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.720711    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-tmp-dir\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.720787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.720729    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-home\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.720787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.720751    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-model-cache\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.722582 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.722558    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-dshm\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.722785 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.722770    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-tls-certs\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.751872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.745784    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfvj8\" (UniqueName: \"kubernetes.io/projected/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-kube-api-access-lfvj8\") pod \"simulator-w3-036184-kserve-7bb7df566-txzvf\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") " pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:15.876104 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:15.876072    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:16.165430 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:16.160711    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"]
Nov 03 19:38:16.373235 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:16.373192    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" event={"ID":"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2","Type":"ContainerStarted","Data":"68d3d5e5908de9bd274bd1ea47bc272e336585284717ba954d3220c92418c80f"}
Nov 03 19:38:16.443056 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:16.442969    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:17.486731 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:17.486698    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:18.320961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:18.320915    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:18.384073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:18.384035    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" event={"ID":"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2","Type":"ContainerStarted","Data":"a4ef52f5a43e4fbee6b6008d295bd130841e17851b76879258181a2c20e56666"}
Nov 03 19:38:18.404393 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:18.404331    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podStartSLOduration=2.0963867 podStartE2EDuration="3.404313466s" totalImagesPullingTime="1.307926766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:18.40376568 +0000 UTC m=+654.658072905" watchObservedRunningTime="2026-11-03 19:38:18.404313466 +0000 UTC m=+654.658620692"
Nov 03 19:38:18.456952 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:18.456920    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:18.676340 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:18.676243    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:18.809089 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:18.809049    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:18.876656 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:18.876626    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:18.950261 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:18.950179    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:19.082781 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:19.082748    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:19.290146 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:19.290106    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:19.548756 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:19.548641    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:19.711501 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:19.711414    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:20.660050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:20.660005    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:21.610620 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:21.610590    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:22.223145 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:22.223110    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:22.914615 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:22.914569    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:23.353295 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.353254    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:23.644450 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.644364    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:23.735170 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.735134    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"]
Nov 03 19:38:23.739919 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.739896    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.747402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.747218    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ef8f84\"/\"kube-root-ca.crt\""
Nov 03 19:38:23.748732 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.748704    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ef8f84\"/\"openshift-service-ca.crt\""
Nov 03 19:38:23.750188 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.750167    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ef8f84\"/\"default-dockercfg-h9vlg\""
Nov 03 19:38:23.750276 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.750194    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ef8f84\"/\"simulator-w0-ef8f84-kserve-self-signed-certs\""
Nov 03 19:38:23.753449 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.753429    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"]
Nov 03 19:38:23.796830 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.796790    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-model-cache\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.797024 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.796873    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-home\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.797024 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.796987    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-tmp-dir\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.797119 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.797100    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-dshm\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.797154 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.797140    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxzm6\" (UniqueName: \"kubernetes.io/projected/c4e11f2e-c05f-4ba6-b748-328c152574a9-kube-api-access-jxzm6\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.797256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.797238    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e11f2e-c05f-4ba6-b748-328c152574a9-tls-certs\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.820884 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.820812    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:23.898155 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.898064    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-dshm\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.898439 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.898403    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxzm6\" (UniqueName: \"kubernetes.io/projected/c4e11f2e-c05f-4ba6-b748-328c152574a9-kube-api-access-jxzm6\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.898563 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.898481    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e11f2e-c05f-4ba6-b748-328c152574a9-tls-certs\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.898631 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.898578    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-model-cache\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.898686 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.898643    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-home\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.898739 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.898678    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-tmp-dir\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.899319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.898874    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-model-cache\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.899319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.899049    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-tmp-dir\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.899319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.899064    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-home\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.900653 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.900637    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-dshm\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.901631 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.901614    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ef8f84\"/\"simulator-w0-ef8f84-kserve-self-signed-certs\""
Nov 03 19:38:23.910994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.910968    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e11f2e-c05f-4ba6-b748-328c152574a9-tls-certs\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.913762 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.913743    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ef8f84\"/\"kube-root-ca.crt\""
Nov 03 19:38:23.926309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.926290    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ef8f84\"/\"openshift-service-ca.crt\""
Nov 03 19:38:23.933726 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.933705    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxzm6\" (UniqueName: \"kubernetes.io/projected/c4e11f2e-c05f-4ba6-b748-328c152574a9-kube-api-access-jxzm6\") pod \"simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") " pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:23.975041 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:23.975005    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:24.067198 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:24.067164    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ef8f84\"/\"default-dockercfg-h9vlg\""
Nov 03 19:38:24.069249 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:24.068248    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:24.112061 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:24.112019    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:24.252140 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:24.252047    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:24.378858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:24.376215    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"]
Nov 03 19:38:24.392321 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:24.392283    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:24.413189 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:24.413163    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" event={"ID":"c4e11f2e-c05f-4ba6-b748-328c152574a9","Type":"ContainerStarted","Data":"6d2670434f5820fdf0a31602a6e493c3f580bc45ee6fcd3dad24155d3d6e48cb"}
Nov 03 19:38:24.565782 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:24.565657    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:25.007061 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:25.006971    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:25.285132 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:25.285098    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:25.426996 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:25.426963    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" event={"ID":"c4e11f2e-c05f-4ba6-b748-328c152574a9","Type":"ContainerStarted","Data":"4339ab245ce4ad4d3a26e1e222f4d9515d112967fc386fa17b37687ed6ea5ea9"}
Nov 03 19:38:25.448107 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:25.448058    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podStartSLOduration=2.448044052 podStartE2EDuration="2.448044052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:25.446619317 +0000 UTC m=+661.700926542" watchObservedRunningTime="2026-11-03 19:38:25.448044052 +0000 UTC m=+661.702351276"
Nov 03 19:38:25.451099 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:25.451068    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:25.583936 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:25.583814    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:25.877290 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:25.877199    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:25.890160 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:25.890135    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:26.068743 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.068706    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:26.081138 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.081107    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"]
Nov 03 19:38:26.085147 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.085123    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.093790 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.093766    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-435bd8\"/\"openshift-service-ca.crt\""
Nov 03 19:38:26.094079 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.094060    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-435bd8\"/\"default-dockercfg-nt8gv\""
Nov 03 19:38:26.094248 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.094092    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-435bd8\"/\"simulator-w2-435bd8-kserve-self-signed-certs\""
Nov 03 19:38:26.096349 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.096317    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-435bd8\"/\"kube-root-ca.crt\""
Nov 03 19:38:26.097816 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.097794    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"]
Nov 03 19:38:26.121827 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.121782    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-dshm\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.122006 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.121848    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb974992-d3ab-4992-8ce4-d639d550163e-tls-certs\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.122051 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.122017    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-home\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.122097 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.122052    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h7sl\" (UniqueName: \"kubernetes.io/projected/cb974992-d3ab-4992-8ce4-d639d550163e-kube-api-access-6h7sl\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.122097 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.122078    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-model-cache\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.122178 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.122104    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-tmp-dir\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.133965 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.133878    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:26.223264 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.223215    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-dshm\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.223468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.223285    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb974992-d3ab-4992-8ce4-d639d550163e-tls-certs\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.223468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.223361    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-home\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.223468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.223395    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h7sl\" (UniqueName: \"kubernetes.io/projected/cb974992-d3ab-4992-8ce4-d639d550163e-kube-api-access-6h7sl\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.223468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.223420    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-model-cache\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.223468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.223442    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-tmp-dir\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.223781 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.223760    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-home\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.223892 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.223868    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-tmp-dir\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.224209 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.224185    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-model-cache\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.226178 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.226155    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-dshm\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.226455 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.226433    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb974992-d3ab-4992-8ce4-d639d550163e-tls-certs\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.243858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.236733    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h7sl\" (UniqueName: \"kubernetes.io/projected/cb974992-d3ab-4992-8ce4-d639d550163e-kube-api-access-6h7sl\") pod \"simulator-w2-435bd8-kserve-56c69bcd9f-zb55b\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") " pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.329062 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.329014    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:26.400063 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.399976    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:26.444332 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.444304    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:38:26.679418 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.679329    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"]
Nov 03 19:38:26.684058 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:38:26.684022    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb974992_d3ab_4992_8ce4_d639d550163e.slice/crio-2ef4f8453a01d51a7bf98f9fcef632e9ae7d507aa6c029ef303c45e9b10e0b5c WatchSource:0}: Error finding container 2ef4f8453a01d51a7bf98f9fcef632e9ae7d507aa6c029ef303c45e9b10e0b5c: Status 404 returned error can't find the container with id 2ef4f8453a01d51a7bf98f9fcef632e9ae7d507aa6c029ef303c45e9b10e0b5c
Nov 03 19:38:26.844100 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.844054    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:26.976506 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:26.976422    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:27.438468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:27.438430    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" event={"ID":"cb974992-d3ab-4992-8ce4-d639d550163e","Type":"ContainerStarted","Data":"2370a74e20e21ca8220c8aba110131865df064d692900e077b953658c863d434"}
Nov 03 19:38:27.438974 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:27.438489    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" event={"ID":"cb974992-d3ab-4992-8ce4-d639d550163e","Type":"ContainerStarted","Data":"2ef4f8453a01d51a7bf98f9fcef632e9ae7d507aa6c029ef303c45e9b10e0b5c"}
Nov 03 19:38:27.460687 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:27.460634    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podStartSLOduration=1.4606161819999999 podStartE2EDuration="1.460616182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:27.457963446 +0000 UTC m=+663.712270738" watchObservedRunningTime="2026-11-03 19:38:27.460616182 +0000 UTC m=+663.714923410"
Nov 03 19:38:27.637450 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:27.637415    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:28.233017 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:28.232964    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:28.400677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:28.400643    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:28.925921 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:28.925683    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:29.075360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.075044    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:29.380448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.379371    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"]
Nov 03 19:38:29.384104 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.384076    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.393709 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.392474    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d8a4ea\"/\"kube-root-ca.crt\""
Nov 03 19:38:29.396696 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.396531    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d8a4ea\"/\"openshift-service-ca.crt\""
Nov 03 19:38:29.398007 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.397983    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d8a4ea\"/\"simulator-w4-d8a4ea-kserve-self-signed-certs\""
Nov 03 19:38:29.398981 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.398957    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"]
Nov 03 19:38:29.431291 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.431233    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d8a4ea\"/\"default-dockercfg-ckb6m\""
Nov 03 19:38:29.465775 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.465742    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-dshm\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.465998 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.465807    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc4wr\" (UniqueName: \"kubernetes.io/projected/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-kube-api-access-hc4wr\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.466061 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.466028    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-model-cache\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.466121 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.466110    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-tmp-dir\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.466218 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.466192    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-tls-certs\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.466267 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.466250    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-home\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.467498 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.467464    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:29.567468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.567435    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4wr\" (UniqueName: \"kubernetes.io/projected/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-kube-api-access-hc4wr\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.567674 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.567475    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-model-cache\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.567674 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.567527    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-tmp-dir\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.567674 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.567559    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-tls-certs\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.567674 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.567582    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-home\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.567674 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.567627    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-dshm\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.568145 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.568115    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-tmp-dir\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.568284 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.568160    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-home\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.568284 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.568198    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-model-cache\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.569897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.569872    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-dshm\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.570228 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.570208    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-tls-certs\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.579503 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.579480    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc4wr\" (UniqueName: \"kubernetes.io/projected/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-kube-api-access-hc4wr\") pod \"simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") " pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:29.610612 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.610563    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:29.710486 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:29.710406    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:30.025040 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:30.025013    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"]
Nov 03 19:38:30.028167 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:38:30.028139    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda39a1945_62e0_42c7_9b2a_d81d2ee042d6.slice/crio-dc5d06f7065f5884b9b83557b3098f5e7156e42c12b612052e13426073583846 WatchSource:0}: Error finding container dc5d06f7065f5884b9b83557b3098f5e7156e42c12b612052e13426073583846: Status 404 returned error can't find the container with id dc5d06f7065f5884b9b83557b3098f5e7156e42c12b612052e13426073583846
Nov 03 19:38:30.358491 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:30.358458    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:30.456137 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:30.456101    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" event={"ID":"a39a1945-62e0-42c7-9b2a-d81d2ee042d6","Type":"ContainerStarted","Data":"9144af6a893056cd8d0099f9fe9a68f086263711f652b51151c8ad83d7a41b34"}
Nov 03 19:38:30.456382 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:30.456149    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" event={"ID":"a39a1945-62e0-42c7-9b2a-d81d2ee042d6","Type":"ContainerStarted","Data":"dc5d06f7065f5884b9b83557b3098f5e7156e42c12b612052e13426073583846"}
Nov 03 19:38:30.476771 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:30.476709    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podStartSLOduration=1.4766909639999999 podStartE2EDuration="1.476690964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:30.474965748 +0000 UTC m=+666.729272973" watchObservedRunningTime="2026-11-03 19:38:30.476690964 +0000 UTC m=+666.730998192"
Nov 03 19:38:30.710910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:30.710818    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:31.131783 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:31.131751    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:31.366885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:31.366826    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:31.945787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:31.945753    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:32.628403 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:32.628373    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:33.325497 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:33.325460    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:33.688736 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:33.688645    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:34.031231 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:34.031137    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:34.068697 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:34.068662    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:34.081264 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:34.081242    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:34.488944 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:34.488915    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:38:34.731750 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:34.731682    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:35.517258 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:35.517217    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:36.142482 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:36.142452    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:36.400769 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:36.400670    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:36.413338 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:36.413308    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:36.498571 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:36.498538    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:38:36.935861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:36.935798    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:37.725363 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:37.725326    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:38.717141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:38.717105    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:39.424973 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:39.424943    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:39.711356 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:39.711260    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:39.724619 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:39.724589    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:40.213536 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:40.213497    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:40.519165 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:40.519090    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:38:40.843443 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:40.843368    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:41.505428 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:41.505392    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:41.916188 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:41.916155    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"]
Nov 03 19:38:41.921754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:41.921727    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:41.930638 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:41.930610    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-036184\"/\"premium-w3-036184-kserve-self-signed-certs\""
Nov 03 19:38:41.933733 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:41.933709    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"]
Nov 03 19:38:41.989139 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:41.989104    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:42.010223 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.010192    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-tls-certs\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.010420 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.010228    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-model-cache\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.010420 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.010311    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-dshm\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.010420 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.010356    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-tmp-dir\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.010592 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.010445    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2rvl\" (UniqueName: \"kubernetes.io/projected/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-kube-api-access-c2rvl\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.010592 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.010562    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-home\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.111829 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.111793    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-tls-certs\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.112060 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.111862    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-model-cache\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.112060 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.111910    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-dshm\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.112060 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.111929    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-tmp-dir\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.112060 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.111974    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2rvl\" (UniqueName: \"kubernetes.io/projected/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-kube-api-access-c2rvl\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.112060 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.112028    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-home\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.112406 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.112379    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-tmp-dir\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.112479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.112424    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-model-cache\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.112479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.112468    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-home\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.114480 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.114448    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-dshm\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.114480 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.114470    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-tls-certs\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.125746 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.125700    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2rvl\" (UniqueName: \"kubernetes.io/projected/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-kube-api-access-c2rvl\") pod \"premium-w3-036184-kserve-8d6bff5d7-mzhxd\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") " pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.138086 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.138053    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:42.237739 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.237655    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:42.526735 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.526654    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"]
Nov 03 19:38:42.539159 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:38:42.539133    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf82e4bbb_88bb_4b6b_b75d_d187d68e3c47.slice/crio-e604af74767db8885df017d9616424bf30d408f254c91643d1c9c198646c8e3d WatchSource:0}: Error finding container e604af74767db8885df017d9616424bf30d408f254c91643d1c9c198646c8e3d: Status 404 returned error can't find the container with id e604af74767db8885df017d9616424bf30d408f254c91643d1c9c198646c8e3d
Nov 03 19:38:42.863020 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:42.862788    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:43.255888 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:38:43.255798    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:38:43.524964 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:43.524865    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" event={"ID":"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47","Type":"ContainerStarted","Data":"e47ba8c6c8130b1e2eec49ce235ccf6ce9f64990d377839c881891e8ee7f4987"}
Nov 03 19:38:43.524964 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:43.524905    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" event={"ID":"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47","Type":"ContainerStarted","Data":"e604af74767db8885df017d9616424bf30d408f254c91643d1c9c198646c8e3d"}
Nov 03 19:38:43.544051 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:43.543982    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podStartSLOduration=2.543964601 podStartE2EDuration="2.543964601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:43.543819817 +0000 UTC m=+679.798127042" watchObservedRunningTime="2026-11-03 19:38:43.543964601 +0000 UTC m=+679.798271827"
Nov 03 19:38:43.610029 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:43.609990    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:44.238716 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:44.238684    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:44.337619 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:44.337533    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:44.990422 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:44.990388    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:45.686500 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:45.686459    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:46.431860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:46.431788    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:47.347223 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:47.347185    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:47.500191 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:47.500157    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:48.029214 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:48.028794    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:48.732022 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:48.731984    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:49.420818 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:49.420783    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:50.017639 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:50.017604    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:50.625721 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:50.625687    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:51.233707 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:51.233665    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:51.602086 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:51.602045    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:51.921878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:51.921751    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:52.204893 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:52.204783    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:52.238143 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:52.238113    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:52.250946 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:52.250921    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:52.580740 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:52.580714    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:38:52.616053 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:52.616019    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:53.312785 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:53.312750    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:53.998651 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:53.998613    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:54.812687 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:54.812647    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:55.380574 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.380538    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"]
Nov 03 19:38:55.387496 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.387445    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.404620 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.404584    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-2c3fc4\"/\"kube-root-ca.crt\""
Nov 03 19:38:55.408196 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.408167    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-2c3fc4\"/\"openshift-service-ca.crt\""
Nov 03 19:38:55.408898 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.408814    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2c3fc4\"/\"premium-w1-2c3fc4-kserve-self-signed-certs\""
Nov 03 19:38:55.409002 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.408918    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2c3fc4\"/\"default-dockercfg-lhd72\""
Nov 03 19:38:55.410393 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.410370    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"]
Nov 03 19:38:55.457983 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.457942    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-model-cache\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.457983 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.457981    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af4841d0-e4fd-4544-8924-cc21f3737772-tls-certs\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.458230 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.458072    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpgr4\" (UniqueName: \"kubernetes.io/projected/af4841d0-e4fd-4544-8924-cc21f3737772-kube-api-access-kpgr4\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.458230 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.458143    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-dshm\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.458230 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.458178    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-home\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.458382 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.458307    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-tmp-dir\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.459027 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.458999    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:55.559580 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.559537    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-dshm\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.559795 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.559598    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-home\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.559795 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.559706    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-tmp-dir\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.559795 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.559768    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-model-cache\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.560011 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.559798    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af4841d0-e4fd-4544-8924-cc21f3737772-tls-certs\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.560011 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.559822    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpgr4\" (UniqueName: \"kubernetes.io/projected/af4841d0-e4fd-4544-8924-cc21f3737772-kube-api-access-kpgr4\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.560121 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.560101    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-home\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.560398 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.560357    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-model-cache\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.561043 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.561021    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-tmp-dir\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.566918 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.566886    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af4841d0-e4fd-4544-8924-cc21f3737772-tls-certs\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.567616 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.567595    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-dshm\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.572991 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.572963    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpgr4\" (UniqueName: \"kubernetes.io/projected/af4841d0-e4fd-4544-8924-cc21f3737772-kube-api-access-kpgr4\") pod \"premium-w1-2c3fc4-kserve-684d46456b-8pcvg\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") " pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.605687 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.605632    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:55.701804 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.701721    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:55.980679 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:55.980593    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"]
Nov 03 19:38:55.984812 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:38:55.984787    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf4841d0_e4fd_4544_8924_cc21f3737772.slice/crio-45405b8e3b3dfe01e302f9437ba1f7a2a14cfd6d94263d49fea07e78580ccc9f WatchSource:0}: Error finding container 45405b8e3b3dfe01e302f9437ba1f7a2a14cfd6d94263d49fea07e78580ccc9f: Status 404 returned error can't find the container with id 45405b8e3b3dfe01e302f9437ba1f7a2a14cfd6d94263d49fea07e78580ccc9f
Nov 03 19:38:56.414397 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:56.414360    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:56.543990 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:56.543953    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:56.603506 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:56.603464    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" event={"ID":"af4841d0-e4fd-4544-8924-cc21f3737772","Type":"ContainerStarted","Data":"07c14163bfef70bcb86543ff904a547f0cb9e5c653447fbdce16778f7384e2ab"}
Nov 03 19:38:56.603506 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:56.603519    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" event={"ID":"af4841d0-e4fd-4544-8924-cc21f3737772","Type":"ContainerStarted","Data":"45405b8e3b3dfe01e302f9437ba1f7a2a14cfd6d94263d49fea07e78580ccc9f"}
Nov 03 19:38:56.627465 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:56.627414    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podStartSLOduration=1.627398104 podStartE2EDuration="1.627398104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:56.624707934 +0000 UTC m=+692.879015160" watchObservedRunningTime="2026-11-03 19:38:56.627398104 +0000 UTC m=+692.881705329"
Nov 03 19:38:56.702294 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:56.702211    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:38:57.061499 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:57.061464    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:57.406170 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:57.406076    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:57.944899 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:57.944872    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:58.643177 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:58.643144    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:58.876203 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:58.876170    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"]
Nov 03 19:38:58.880111 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:58.880085    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:58.890763 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:58.890734    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-05e217\"/\"openshift-service-ca.crt\""
Nov 03 19:38:58.890929 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:58.890765    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-05e217\"/\"premium-w5-05e217-kserve-self-signed-certs\""
Nov 03 19:38:58.890929 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:58.890820    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-05e217\"/\"kube-root-ca.crt\""
Nov 03 19:38:58.890929 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:58.890867    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-05e217\"/\"default-dockercfg-p65gc\""
Nov 03 19:38:58.892919 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:58.892893    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"]
Nov 03 19:38:58.904073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:58.903985    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-model-cache\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:58.904213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:58.904075    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-dshm\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:58.904213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:58.904149    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg7g7\" (UniqueName: \"kubernetes.io/projected/fae76152-1c87-4685-8074-6bcc1760268c-kube-api-access-bg7g7\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:58.904213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:58.904170    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-home\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:58.904213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:58.904194    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-tmp-dir\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:58.904420 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:58.904267    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fae76152-1c87-4685-8074-6bcc1760268c-tls-certs\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:58.965637 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:58.965607    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:59.005100 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.005065    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-model-cache\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:59.005297 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.005127    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-dshm\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:59.005297 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.005167    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg7g7\" (UniqueName: \"kubernetes.io/projected/fae76152-1c87-4685-8074-6bcc1760268c-kube-api-access-bg7g7\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:59.005297 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.005185    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-home\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:59.005297 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.005205    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-tmp-dir\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:59.005297 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.005291    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fae76152-1c87-4685-8074-6bcc1760268c-tls-certs\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:59.005631 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.005600    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-model-cache\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:59.005631 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.005620    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-home\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:59.005851 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.005807    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-tmp-dir\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:59.007630 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.007575    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-dshm\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:59.008291 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.008248    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fae76152-1c87-4685-8074-6bcc1760268c-tls-certs\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:59.022883 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.022798    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg7g7\" (UniqueName: \"kubernetes.io/projected/fae76152-1c87-4685-8074-6bcc1760268c-kube-api-access-bg7g7\") pod \"premium-w5-05e217-kserve-69469cd78f-bqvzv\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") " pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:59.194500 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.194413    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:38:59.447572 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.447464    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:59.494361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.494325    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"]
Nov 03 19:38:59.621726 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.621693    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" event={"ID":"fae76152-1c87-4685-8074-6bcc1760268c","Type":"ContainerStarted","Data":"a38a4a4acdfcc5b146d077911a741036ccc3e5c7fab00970f2545f1c12858df4"}
Nov 03 19:38:59.621913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.621735    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" event={"ID":"fae76152-1c87-4685-8074-6bcc1760268c","Type":"ContainerStarted","Data":"f83d00ed835e8b929471c2a520294650bcde2c637bed9eab8a9d46bbc82e5ff0"}
Nov 03 19:38:59.643421 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:38:59.643364    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podStartSLOduration=1.643348886 podStartE2EDuration="1.643348886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:59.641211311 +0000 UTC m=+695.895518540" watchObservedRunningTime="2026-11-03 19:38:59.643348886 +0000 UTC m=+695.897656110"
Nov 03 19:39:00.119972 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:00.119937    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:00.194522 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:00.194484    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:39:00.729039 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:00.729004    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:01.416925 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:01.416885    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:02.227947 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:02.227911    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:02.940636 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:02.940601    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:03.628279 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:03.628230    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:04.229993 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:04.229957    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:04.889474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:04.889428    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:05.542607 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:05.542573    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:05.702438 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:05.702399    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:39:05.721408 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:05.721382    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:39:06.232357 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:06.232321    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:06.681712 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:06.681678    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:39:06.828526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:06.828493    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:07.522199 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:07.522169    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:08.330335 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:08.330296    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:09.097077 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:09.097041    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:09.194697 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:09.194664    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:39:09.207276 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:09.207252    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:39:09.692606 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:09.692579    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:39:09.829524 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:09.829484    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:10.540133 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:10.539586    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:11.234340 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:11.234302    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:11.821891 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:11.821362    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:12.420485 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:12.420458    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:13.211230 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:13.211195    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:13.273706 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:39:13.273670    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:39:14.307277 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:14.307240    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:15.027847 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:15.027506    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:15.766087 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:15.762427    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:16.624574 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:16.624540    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:17.345173 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:17.345135    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:18.219805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:18.219771    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:18.950917 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:18.949160    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:19.636110 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:19.636076    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:20.238752 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:20.238586    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:20.829012 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:20.828975    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:21.487959 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:21.487083    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:22.192301 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:22.192263    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:22.811628 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:22.811585    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:23.419577 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:23.419539    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:24.018431 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:24.018400    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:24.735285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:24.735235    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:25.569319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:25.569254    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:26.324244 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:26.324212    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:26.925746 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:26.925716    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:27.620185 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:27.620149    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:28.236310 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:28.236278    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:28.919774 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:28.919734    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:29.171195 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:29.171096    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:29.691418 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:29.691371    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:30.326564 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:30.326531    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:31.006236 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:31.006199    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:31.622681 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:31.622646    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:32.513074 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:32.513037    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:33.303611 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:33.303580    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:34.065123 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:34.065087    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:34.989232 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:34.989145    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:35.930119 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:35.929852    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:36.898212 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:36.898174    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:37.729380 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:37.729346    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:38.505857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:38.503775    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:39.110263 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:39.110225    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:39.830461 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:39.830424    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:40.538905 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:40.538638    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:41.123794 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:41.123763    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:41.718182 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:41.718152    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:42.320178 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:42.320145    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:42.899393 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:42.899334    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:43.051063 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:43.051017    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:43.255041 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:39:43.254964    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:39:43.750503 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:43.748895    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:44.440310 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:44.440278    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:45.133858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:45.133799    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:46.016730 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:46.016688    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:46.631232 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:46.631196    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:47.362400 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:47.362366    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:48.052985 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:48.052946    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:48.625618 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:48.625577    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:49.309773 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:49.309735    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:50.029177 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:50.029013    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:50.764332 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:50.764294    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:51.459036 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:51.458992    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:52.158858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:52.153479    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:52.830084 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:52.830037    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:53.438928 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:53.438893    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:54.056857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.056039    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:54.057364 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.057328    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"]
Nov 03 19:39:54.075681 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.075463    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"
Nov 03 19:39:54.078452 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.075370    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"]
Nov 03 19:39:54.081568 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.081544    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:39:54.098475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.083137    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\""
Nov 03 19:39:54.229010 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.228976    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cef245bb-72q42\" (UID: \"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"
Nov 03 19:39:54.229190 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.229020    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8z85\" (UniqueName: \"kubernetes.io/projected/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-kube-api-access-h8z85\") pod \"maas-api-revoke-sub-cef245bb-72q42\" (UID: \"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"
Nov 03 19:39:54.229190 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.229150    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cef245bb-72q42\" (UID: \"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"
Nov 03 19:39:54.330313 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.330236    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cef245bb-72q42\" (UID: \"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"
Nov 03 19:39:54.330470 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.330318    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cef245bb-72q42\" (UID: \"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"
Nov 03 19:39:54.330470 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.330341    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8z85\" (UniqueName: \"kubernetes.io/projected/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-kube-api-access-h8z85\") pod \"maas-api-revoke-sub-cef245bb-72q42\" (UID: \"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"
Nov 03 19:39:54.331033 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.331008    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cef245bb-72q42\" (UID: \"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"
Nov 03 19:39:54.332682 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.332651    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cef245bb-72q42\" (UID: \"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"
Nov 03 19:39:54.344283 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.344257    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8z85\" (UniqueName: \"kubernetes.io/projected/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-kube-api-access-h8z85\") pod \"maas-api-revoke-sub-cef245bb-72q42\" (UID: \"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"
Nov 03 19:39:54.402296 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.402263    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"
Nov 03 19:39:54.471166 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.469127    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"]
Nov 03 19:39:54.476712 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.476691    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.483530 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.483507    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-05e217-openshift-default-dockercfg-r9knx\""
Nov 03 19:39:54.496028 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.490175    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"]
Nov 03 19:39:54.632918 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.632810    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.632918 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.632888    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.632918 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.632913    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.633178 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.632938    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.633178 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.632968    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704a8596-0c3b-403f-b847-7ce2256eb13d-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.633178 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.633033    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.633178 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.633095    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9c52\" (UniqueName: \"kubernetes.io/projected/704a8596-0c3b-403f-b847-7ce2256eb13d-kube-api-access-w9c52\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.633367 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.633183    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.633367 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.633235    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.678890 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.678866    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"]
Nov 03 19:39:54.681825 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:39:54.681780    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd0cc75_1454_4fe5_a0b9_9fbdddbd04fb.slice/crio-fe9f2b6546cde7704a4bbfad31fb56ca940c157ebae0f7a16cc5d443a7b8fcb9 WatchSource:0}: Error finding container fe9f2b6546cde7704a4bbfad31fb56ca940c157ebae0f7a16cc5d443a7b8fcb9: Status 404 returned error can't find the container with id fe9f2b6546cde7704a4bbfad31fb56ca940c157ebae0f7a16cc5d443a7b8fcb9
Nov 03 19:39:54.721338 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.721301    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:54.747722 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.747677    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.747960 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.747749    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.747960 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.747790    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.747960 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.747824    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.747960 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.747871    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704a8596-0c3b-403f-b847-7ce2256eb13d-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.747960 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.747907    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.747960 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.747933    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9c52\" (UniqueName: \"kubernetes.io/projected/704a8596-0c3b-403f-b847-7ce2256eb13d-kube-api-access-w9c52\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.748654 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.747995    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.748654 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.748032    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.748654 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.748314    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.748654 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.748347    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.748891 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.748776    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.748931 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.748905    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.750464 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.749291    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704a8596-0c3b-403f-b847-7ce2256eb13d-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.751764 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.751727    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.751932 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.751797    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.761139 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.761115    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.761425 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.761400    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9c52\" (UniqueName: \"kubernetes.io/projected/704a8596-0c3b-403f-b847-7ce2256eb13d-kube-api-access-w9c52\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.801371 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.801338    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:54.918217 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:54.918135    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42" event={"ID":"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb","Type":"ContainerStarted","Data":"fe9f2b6546cde7704a4bbfad31fb56ca940c157ebae0f7a16cc5d443a7b8fcb9"}
Nov 03 19:39:55.050557 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.050526    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"]
Nov 03 19:39:55.054369 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.054339    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.059710 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.059690    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-c5f6cb-openshift-default-dockercfg-p6txp\""
Nov 03 19:39:55.070344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.070318    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"]
Nov 03 19:39:55.100153 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.100129    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"]
Nov 03 19:39:55.104731 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:39:55.104704    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704a8596_0c3b_403f_b847_7ce2256eb13d.slice/crio-bbcffa68573b5fea1af7fe458011a3db6885f512a91b0abd20da8b78cb52aa1e WatchSource:0}: Error finding container bbcffa68573b5fea1af7fe458011a3db6885f512a91b0abd20da8b78cb52aa1e: Status 404 returned error can't find the container with id bbcffa68573b5fea1af7fe458011a3db6885f512a91b0abd20da8b78cb52aa1e
Nov 03 19:39:55.106321 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.106289    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:39:55.106423 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.106367    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:39:55.106423 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.106406    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:39:55.152626 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.152600    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.152740 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.152651    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.152740 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.152723    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.152851 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.152754    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn4fs\" (UniqueName: \"kubernetes.io/projected/0398a799-4e11-4f8e-913f-8a42574c0ae9-kube-api-access-dn4fs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.152851 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.152793    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.153015 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.152994    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.153066 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.153031    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.153142 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.153122    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0398a799-4e11-4f8e-913f-8a42574c0ae9-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.153215 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.153198    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.254064 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.253969    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn4fs\" (UniqueName: \"kubernetes.io/projected/0398a799-4e11-4f8e-913f-8a42574c0ae9-kube-api-access-dn4fs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.254064 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.254054    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.254296 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.254122    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.254296 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.254146    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.254296 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.254184    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0398a799-4e11-4f8e-913f-8a42574c0ae9-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.254296 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.254241    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.254524 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.254292    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.254524 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.254362    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.254524 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.254382    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.254891 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.254858    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.254988 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.254923    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.255124 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.255098    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.255429 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.255402    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0398a799-4e11-4f8e-913f-8a42574c0ae9-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.255534 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.255483    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.256930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.256909    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.257134 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.257113    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.266529 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.266478    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.266672 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.266590    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn4fs\" (UniqueName: \"kubernetes.io/projected/0398a799-4e11-4f8e-913f-8a42574c0ae9-kube-api-access-dn4fs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.369496 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.369465    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:55.582633 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.582598    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:55.667102 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.667065    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"]
Nov 03 19:39:55.672413 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:39:55.672384    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0398a799_4e11_4f8e_913f_8a42574c0ae9.slice/crio-7efdda866eb53c22200cf6efb3343297eecbaafb050fbcdf71c5662cec0d12a5 WatchSource:0}: Error finding container 7efdda866eb53c22200cf6efb3343297eecbaafb050fbcdf71c5662cec0d12a5: Status 404 returned error can't find the container with id 7efdda866eb53c22200cf6efb3343297eecbaafb050fbcdf71c5662cec0d12a5
Nov 03 19:39:55.675350 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.675320    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:39:55.675477 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.675390    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:39:55.675477 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.675419    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:39:55.926366 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.926287    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw" event={"ID":"704a8596-0c3b-403f-b847-7ce2256eb13d","Type":"ContainerStarted","Data":"7cb82223e95f22f85a75a443e366aae304db9aa20ff14c35fd298b2bf01a9870"}
Nov 03 19:39:55.926366 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.926359    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw" event={"ID":"704a8596-0c3b-403f-b847-7ce2256eb13d","Type":"ContainerStarted","Data":"bbcffa68573b5fea1af7fe458011a3db6885f512a91b0abd20da8b78cb52aa1e"}
Nov 03 19:39:55.928518 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.928485    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk" event={"ID":"0398a799-4e11-4f8e-913f-8a42574c0ae9","Type":"ContainerStarted","Data":"d72f758659f5ff0a21e09c6338854b7067c9ba2c5a4aeb20101eb6cffeb53402"}
Nov 03 19:39:55.928655 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.928539    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk" event={"ID":"0398a799-4e11-4f8e-913f-8a42574c0ae9","Type":"ContainerStarted","Data":"7efdda866eb53c22200cf6efb3343297eecbaafb050fbcdf71c5662cec0d12a5"}
Nov 03 19:39:55.953573 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:55.953382    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw" podStartSLOduration=1.9533649789999998 podStartE2EDuration="1.953364979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:55.951981133 +0000 UTC m=+752.206288358" watchObservedRunningTime="2026-11-03 19:39:55.953364979 +0000 UTC m=+752.207672205"
Nov 03 19:39:56.000860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:56.000652    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk" podStartSLOduration=1.000636978 podStartE2EDuration="1.000636978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:55.99922652 +0000 UTC m=+752.253533746" watchObservedRunningTime="2026-11-03 19:39:56.000636978 +0000 UTC m=+752.254944213"
Nov 03 19:39:56.344448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:56.344409    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:56.385801 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:56.381204    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:39:56.385801 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:56.382871    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 19:39:56.385801 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:56.382925    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk" podUID="0398a799-4e11-4f8e-913f-8a42574c0ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 19:39:56.818530 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:56.818186    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:39:56.819863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:56.819802    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 19:39:56.820001 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:56.819886    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw" podUID="704a8596-0c3b-403f-b847-7ce2256eb13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 19:39:57.241419 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:57.241333    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:57.401860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:57.391305    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 19:39:57.401860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:57.391367    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk" podUID="0398a799-4e11-4f8e-913f-8a42574c0ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 19:39:57.806749 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:57.806281    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 19:39:57.806749 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:57.806347    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw" podUID="704a8596-0c3b-403f-b847-7ce2256eb13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 19:39:58.045791 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:58.045751    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:58.371862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:58.370055    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 19:39:58.371862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:58.370141    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk" podUID="0398a799-4e11-4f8e-913f-8a42574c0ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 19:39:58.806912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:58.802389    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 19:39:58.806912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:58.802444    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw" podUID="704a8596-0c3b-403f-b847-7ce2256eb13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 19:39:58.946001 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:58.945959    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:59.385868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:59.384566    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 19:39:59.385868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:59.384630    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk" podUID="0398a799-4e11-4f8e-913f-8a42574c0ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 19:39:59.805343 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:59.805304    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 19:39:59.805617 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:59.805589    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw" podUID="704a8596-0c3b-403f-b847-7ce2256eb13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 19:39:59.809889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:39:59.806556    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:00.381283 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:00.381242    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 19:40:00.381576 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:00.381539    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk" podUID="0398a799-4e11-4f8e-913f-8a42574c0ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 19:40:00.623311 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:00.623273    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:00.805276 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:00.805234    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 19:40:00.805579 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:00.805552    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw" podUID="704a8596-0c3b-403f-b847-7ce2256eb13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 19:40:00.959121 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:00.959088    5237 generic.go:356] "Generic (PLEG): container finished" podID="1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb" containerID="204031757d4e5cb5ae27aa4195eab67c965b77a8d59466746262f52110259537" exitCode=0
Nov 03 19:40:00.959502 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:00.959225    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42" event={"ID":"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb","Type":"ContainerDied","Data":"204031757d4e5cb5ae27aa4195eab67c965b77a8d59466746262f52110259537"}
Nov 03 19:40:01.372273 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:01.372232    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 19:40:01.372584 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:01.372552    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk" podUID="0398a799-4e11-4f8e-913f-8a42574c0ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 19:40:01.456998 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:01.456959    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:01.801964 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:01.801928    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 19:40:01.802159 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:01.802007    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw" podUID="704a8596-0c3b-403f-b847-7ce2256eb13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 19:40:02.166710 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:02.166677    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"
Nov 03 19:40:02.234500 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:02.234453    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:02.246987 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:02.246949    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-maas-api-xks-ca\") pod \"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb\" (UID: \"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb\") "
Nov 03 19:40:02.247169 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:02.247147    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-kube-api-access-h8z85\" (UniqueName: \"kubernetes.io/projected/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-kube-api-access-h8z85\") pod \"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb\" (UID: \"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb\") "
Nov 03 19:40:02.247223 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:02.247190    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-maas-api-openshift-ca\") pod \"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb\" (UID: \"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb\") "
Nov 03 19:40:02.247633 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:02.247596    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-maas-api-openshift-ca" pod "1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb" (UID: "1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:40:02.249270 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:02.249240    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-maas-api-xks-ca" pod "1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb" (UID: "1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:02.249468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:02.249451    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-kube-api-access-h8z85" pod "1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb" (UID: "1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb"). InnerVolumeSpecName "kube-api-access-h8z85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:40:02.349388 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:02.349298    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8z85\" (UniqueName: \"kubernetes.io/projected/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-kube-api-access-h8z85\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:02.349388 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:02.349332    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-maas-api-openshift-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:02.349388 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:02.349346    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb-maas-api-xks-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:02.370168 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:02.370128    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 19:40:02.370335 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:02.370192    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk" podUID="0398a799-4e11-4f8e-913f-8a42574c0ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 19:40:02.943236 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:02.943194    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:02.973654 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:02.973623    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"
Nov 03 19:40:02.973876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:02.973679    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42" event={"ID":"1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb","Type":"ContainerDied","Data":"fe9f2b6546cde7704a4bbfad31fb56ca940c157ebae0f7a16cc5d443a7b8fcb9"}
Nov 03 19:40:02.973876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:02.973743    5237 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe9f2b6546cde7704a4bbfad31fb56ca940c157ebae0f7a16cc5d443a7b8fcb9"
Nov 03 19:40:03.064700 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:03.064668    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:40:03.065154 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:03.065128    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:40:03.065989 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:03.065966    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:40:03.703147 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:03.703110    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:03.761641 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:03.761612    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:40:03.762491 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:03.762471    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:40:03.763608 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:03.763583    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:40:04.513263 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:04.513225    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:05.204101 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:05.204065    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:06.218152 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:06.218088    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:06.935157 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:06.935119    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:07.825969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:07.825928    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:08.421560 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:08.421521    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:09.036145 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:09.036109    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:09.636823 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:09.636786    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:10.436654 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:10.436617    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:11.140729 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:11.140696    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:11.922410 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:11.922373    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:12.612010 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:12.611970    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:13.274895 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:40:13.274854    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:40:13.838917 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:13.838881    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:14.859179 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:14.859146    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:15.885528 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:15.885492    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:16.619260 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:16.619221    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:17.213001 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:17.212962    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:17.829690 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:17.829659    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:18.710737 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:18.710699    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:18.787421 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:18.787384    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"]
Nov 03 19:40:18.788096 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:18.788073    5237 cpu_manager.go:414] "containerMap: removing container" podUID="1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb" containerName="revoke-keys"
Nov 03 19:40:18.788204 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:18.788099    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb" containerName="revoke-keys"
Nov 03 19:40:18.788323 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:18.788303    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb" containerName="revoke-keys"
Nov 03 19:40:18.792081 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:18.792058    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:18.804367 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:18.804334    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"]
Nov 03 19:40:18.805302 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:18.804655    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d8a4ea\"/\"distinct-w4-d8a4ea-kserve-self-signed-certs\""
Nov 03 19:40:18.873926 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:18.873889    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:18.895885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:18.895846    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-model-cache\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:18.896057 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:18.895932    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jbb9\" (UniqueName: \"kubernetes.io/projected/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-kube-api-access-5jbb9\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:18.896057 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:18.895968    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-home\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:18.896057 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:18.896038    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-dshm\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:18.896178 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:18.896116    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-tls-certs\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:18.896245 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:18.896231    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-tmp-dir\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:19.004510 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:19.004420    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-dshm\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:19.005176 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:19.005140    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-tls-certs\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:19.005438 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:19.005421    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-tmp-dir\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:19.006004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:19.005985    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-model-cache\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:19.006554 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:19.006535    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jbb9\" (UniqueName: \"kubernetes.io/projected/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-kube-api-access-5jbb9\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:19.007093 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:19.007074    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-home\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:19.007556 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:19.006462    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-model-cache\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:19.007730 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:19.007449    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-home\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:19.007871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:19.005883    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-tmp-dir\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:19.014856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:19.013667    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-dshm\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:19.025895 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:19.020340    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-tls-certs\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:19.025895 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:19.025122    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jbb9\" (UniqueName: \"kubernetes.io/projected/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-kube-api-access-5jbb9\") pod \"distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") " pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:19.110431 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:19.110395    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:19.392886 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:19.392856    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"]
Nov 03 19:40:19.396181 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:40:19.396157    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a61b39a_3f50_4ff7_88fe_dc8f851db36b.slice/crio-2c930c6bcb0d93b213d8bcd08ea1f0e7bc222ed4031c5c2d4cfd7d410974d530 WatchSource:0}: Error finding container 2c930c6bcb0d93b213d8bcd08ea1f0e7bc222ed4031c5c2d4cfd7d410974d530: Status 404 returned error can't find the container with id 2c930c6bcb0d93b213d8bcd08ea1f0e7bc222ed4031c5c2d4cfd7d410974d530
Nov 03 19:40:19.856904 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:19.856866    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:20.076003 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:20.075951    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" event={"ID":"1a61b39a-3f50-4ff7-88fe-dc8f851db36b","Type":"ContainerStarted","Data":"aa1524641693abd488840945ccefd3e4bd7989823c7c184606cb3efaa31e7b63"}
Nov 03 19:40:20.076003 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:20.076007    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" event={"ID":"1a61b39a-3f50-4ff7-88fe-dc8f851db36b","Type":"ContainerStarted","Data":"2c930c6bcb0d93b213d8bcd08ea1f0e7bc222ed4031c5c2d4cfd7d410974d530"}
Nov 03 19:40:20.097559 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:20.097506    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podStartSLOduration=2.097491928 podStartE2EDuration="2.097491928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:20.095897721 +0000 UTC m=+776.350204956" watchObservedRunningTime="2026-11-03 19:40:20.097491928 +0000 UTC m=+776.351799152"
Nov 03 19:40:20.111002 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:20.110916    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:20.536245 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:20.536211    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:21.309517 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:21.309476    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:21.933446 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:21.933410    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:22.640976 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:22.640937    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:23.330333 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:23.330298    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:24.519638 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:24.518963    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:24.875689 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:24.875655    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"]
Nov 03 19:40:24.879601 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:24.879585    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:24.884992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:24.884962    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-2c3fc4-openshift-default-dockercfg-f8v5n\""
Nov 03 19:40:24.894514 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:24.894489    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"]
Nov 03 19:40:24.967081 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:24.967043    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4a25ae-6965-4713-ab67-327317f678d8-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:24.967277 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:24.967098    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:24.967277 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:24.967116    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:24.967361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:24.967270    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6grv\" (UniqueName: \"kubernetes.io/projected/2b4a25ae-6965-4713-ab67-327317f678d8-kube-api-access-z6grv\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:24.967361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:24.967327    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b4a25ae-6965-4713-ab67-327317f678d8-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:24.967442 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:24.967379    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4a25ae-6965-4713-ab67-327317f678d8-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:24.967442 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:24.967407    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:24.967524 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:24.967442    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:24.967524 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:24.967465    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.068243 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.068209    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b4a25ae-6965-4713-ab67-327317f678d8-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.068428 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.068257    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4a25ae-6965-4713-ab67-327317f678d8-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.068428 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.068281    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.068428 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.068324    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.068428 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.068374    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.068428 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.068413    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4a25ae-6965-4713-ab67-327317f678d8-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.068706 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.068463    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.068706 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.068486    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.068706 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.068538    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6grv\" (UniqueName: \"kubernetes.io/projected/2b4a25ae-6965-4713-ab67-327317f678d8-kube-api-access-z6grv\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.068890 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.068808    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.069199 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.069095    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.069199 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.069108    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4a25ae-6965-4713-ab67-327317f678d8-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.069199 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.069142    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.069390 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.069263    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.071277 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.071244    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.071578 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.071557    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4a25ae-6965-4713-ab67-327317f678d8-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.081281 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.081251    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b4a25ae-6965-4713-ab67-327317f678d8-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.081480 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.081462    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6grv\" (UniqueName: \"kubernetes.io/projected/2b4a25ae-6965-4713-ab67-327317f678d8-kube-api-access-z6grv\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.193214 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.193114    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:25.227228 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.227196    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:25.505294 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.505220    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"]
Nov 03 19:40:25.508959 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:40:25.508924    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4a25ae_6965_4713_ab67_327317f678d8.slice/crio-bd18bb4918ed919fd6fc8272eab5c5fea29ec62a01ad7a832c405ae80ac151d8 WatchSource:0}: Error finding container bd18bb4918ed919fd6fc8272eab5c5fea29ec62a01ad7a832c405ae80ac151d8: Status 404 returned error can't find the container with id bd18bb4918ed919fd6fc8272eab5c5fea29ec62a01ad7a832c405ae80ac151d8
Nov 03 19:40:25.511744 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.511713    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:40:25.511880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.511774    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:40:25.511880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:25.511801    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:40:26.107669 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:26.107633    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm" event={"ID":"2b4a25ae-6965-4713-ab67-327317f678d8","Type":"ContainerStarted","Data":"7208ce730ab4cc3e006c968ed14b4def81f8c01b75fb3e5aad77dc714cd1b475"}
Nov 03 19:40:26.108195 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:26.107699    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm" event={"ID":"2b4a25ae-6965-4713-ab67-327317f678d8","Type":"ContainerStarted","Data":"bd18bb4918ed919fd6fc8272eab5c5fea29ec62a01ad7a832c405ae80ac151d8"}
Nov 03 19:40:26.147859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:26.138372    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm" podStartSLOduration=2.138349257 podStartE2EDuration="2.138349257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:26.131754581 +0000 UTC m=+782.386061818" watchObservedRunningTime="2026-11-03 19:40:26.138349257 +0000 UTC m=+782.392656482"
Nov 03 19:40:26.147859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:26.139347    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:26.194371 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:26.194334    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:26.195807 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:26.195785    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 19:40:26.195912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:26.195860    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm" podUID="2b4a25ae-6965-4713-ab67-327317f678d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 19:40:26.900863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:26.889419    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:27.194969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:27.194311    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 19:40:27.194969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:27.194373    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm" podUID="2b4a25ae-6965-4713-ab67-327317f678d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 19:40:27.714222 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:27.714184    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:28.194306 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:28.194266    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 19:40:28.194520 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:28.194333    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm" podUID="2b4a25ae-6965-4713-ab67-327317f678d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 19:40:28.563266 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:28.563226    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:29.128858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:29.125019    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:29.202871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:29.200972    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 19:40:29.202871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:29.201031    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm" podUID="2b4a25ae-6965-4713-ab67-327317f678d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 19:40:29.258984 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:29.257937    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:29.659264 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:29.659145    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:29.814771 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:29.814734    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:30.197331 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:30.197293    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 19:40:30.197637 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:30.197601    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm" podUID="2b4a25ae-6965-4713-ab67-327317f678d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 19:40:30.335585 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:30.335556    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:40:30.461563 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:30.461423    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:31.219862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:31.212281    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 19:40:31.219862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:31.212361    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm" podUID="2b4a25ae-6965-4713-ab67-327317f678d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 19:40:31.585857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:31.585391    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:32.196860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:32.195695    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 19:40:32.196860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:32.195819    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm" podUID="2b4a25ae-6965-4713-ab67-327317f678d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 19:40:32.685237 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:32.685190    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:33.351377 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:33.351342    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:33.746774 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:33.746688    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:33.747254 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:33.747162    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:33.748114 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:33.748089    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:40:34.116300 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:34.116245    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:34.845085 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:34.844997    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:35.492434 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:35.492109    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:36.888162 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:36.888130    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:37.925117 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:37.925078    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:38.736021 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:38.735984    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:39.125295 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.125259    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:39.464373 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.464292    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"]
Nov 03 19:40:39.468135 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.468109    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.473074 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.473054    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-036184-openshift-default-dockercfg-xkkwv\""
Nov 03 19:40:39.482466 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.482441    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"]
Nov 03 19:40:39.565770 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.565709    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"]
Nov 03 19:40:39.597859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.594939    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:39.597859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.595579    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.597859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.595604    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k6gn\" (UniqueName: \"kubernetes.io/projected/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-kube-api-access-4k6gn\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.597859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.595639    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.597859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.595669    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.597859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.595736    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.597859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.595763    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.597859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.595783    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.597859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.595806    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.597859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.595855    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.613464 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.613435    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"]
Nov 03 19:40:39.613777 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.613762    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.696777 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.696741    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.696988 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.696780    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.696988 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.696888    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.696988 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.696910    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.696988 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.696940    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.696988 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.696974    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.697309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.697012    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.697309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.697097    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.697309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.697127    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k6gn\" (UniqueName: \"kubernetes.io/projected/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-kube-api-access-4k6gn\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.697309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.697234    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.697494 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.697384    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.697610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.697564    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.697746 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.697623    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.698202 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.698174    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.699344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.699321    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.699487 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.699469    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.709716 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.709688    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k6gn\" (UniqueName: \"kubernetes.io/projected/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-kube-api-access-4k6gn\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.710363 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.710343    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.782985 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.782908    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:39.798043 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.797998    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nw9p\" (UniqueName: \"kubernetes.io/projected/9fea1109-4466-401f-8de1-3c9fae530f70-kube-api-access-7nw9p\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.798124 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.798043    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fea1109-4466-401f-8de1-3c9fae530f70-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.798124 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.798065    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.798265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.798238    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fea1109-4466-401f-8de1-3c9fae530f70-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.798330 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.798298    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.798410 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.798385    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.798474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.798464    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fea1109-4466-401f-8de1-3c9fae530f70-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.798549 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.798534    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.798605 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.798588    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.899517 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.899464    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fea1109-4466-401f-8de1-3c9fae530f70-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.899717 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.899569    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.899717 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.899596    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.899717 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.899669    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nw9p\" (UniqueName: \"kubernetes.io/projected/9fea1109-4466-401f-8de1-3c9fae530f70-kube-api-access-7nw9p\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.899717 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.899703    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fea1109-4466-401f-8de1-3c9fae530f70-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.901050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.899735    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.901050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.899783    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fea1109-4466-401f-8de1-3c9fae530f70-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.901050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.899810    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.901050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.899875    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.901050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.900383    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.901050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.900412    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.901050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.900632    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.901050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.900665    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.901050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.900900    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fea1109-4466-401f-8de1-3c9fae530f70-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.902235 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.902207    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.902896 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.902870    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fea1109-4466-401f-8de1-3c9fae530f70-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.911960 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.911938    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fea1109-4466-401f-8de1-3c9fae530f70-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.912197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.912172    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nw9p\" (UniqueName: \"kubernetes.io/projected/9fea1109-4466-401f-8de1-3c9fae530f70-kube-api-access-7nw9p\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:39.932089 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:39.932053    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:40.071172 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.071137    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"]
Nov 03 19:40:40.074543 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:40:40.073368    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a4dcf8_18ac_443b_b78b_014a4e38ce34.slice/crio-b36774d446631d5b4ed10bbdc15eee6a09bbbe96de2361d3c6c7f9b5bd4d500f WatchSource:0}: Error finding container b36774d446631d5b4ed10bbdc15eee6a09bbbe96de2361d3c6c7f9b5bd4d500f: Status 404 returned error can't find the container with id b36774d446631d5b4ed10bbdc15eee6a09bbbe96de2361d3c6c7f9b5bd4d500f
Nov 03 19:40:40.077324 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.077291    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:40:40.077425 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.077353    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:40:40.077425 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.077381    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:40:40.217856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.217205    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"]
Nov 03 19:40:40.221397 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.221373    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.243035 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.243008    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d8a4ea-openshift-default-dockercfg-wkq5q\""
Nov 03 19:40:40.248148 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.248119    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"]
Nov 03 19:40:40.307862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.307321    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"]
Nov 03 19:40:40.309210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.308654    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm" event={"ID":"e6a4dcf8-18ac-443b-b78b-014a4e38ce34","Type":"ContainerStarted","Data":"ee6f02cf22caf9e58e6e26855329365b5065a44226c9605f0b0dc53f6c118e8f"}
Nov 03 19:40:40.309210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.308705    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm" event={"ID":"e6a4dcf8-18ac-443b-b78b-014a4e38ce34","Type":"ContainerStarted","Data":"b36774d446631d5b4ed10bbdc15eee6a09bbbe96de2361d3c6c7f9b5bd4d500f"}
Nov 03 19:40:40.312363 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.312335    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.312492 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.312385    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.312492 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.312415    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj9xx\" (UniqueName: \"kubernetes.io/projected/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-kube-api-access-gj9xx\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.312492 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.312478    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.312667 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.312515    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.312667 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.312613    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.312770 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.312725    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.312770 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.312759    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.312938 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.312817    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.325821 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:40:40.325786    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fea1109_4466_401f_8de1_3c9fae530f70.slice/crio-7b7caa6101499b58e18d0cc3d2bcb3d4792e8e3455c58758496547eb32e9acf1 WatchSource:0}: Error finding container 7b7caa6101499b58e18d0cc3d2bcb3d4792e8e3455c58758496547eb32e9acf1: Status 404 returned error can't find the container with id 7b7caa6101499b58e18d0cc3d2bcb3d4792e8e3455c58758496547eb32e9acf1
Nov 03 19:40:40.329025 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.328977    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:40:40.329147 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.329057    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:40:40.329147 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.329104    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:40:40.411810 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.411738    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm" podStartSLOduration=1.411716817 podStartE2EDuration="1.411716817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:40.404361275 +0000 UTC m=+796.658668500" watchObservedRunningTime="2026-11-03 19:40:40.411716817 +0000 UTC m=+796.666024042"
Nov 03 19:40:40.414214 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.414185    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.414461 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.414444    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.415632 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.415609    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.416128 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.416107    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.418016 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.417996    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.418484 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.418467    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.418977 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.418948    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj9xx\" (UniqueName: \"kubernetes.io/projected/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-kube-api-access-gj9xx\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.419168 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.419150    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.419319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.419302    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.419772 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.418891    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.419939 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.415350    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.420252 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.418389    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.420368 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.419633    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.420475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.416885    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.426558 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.426532    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.426957 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.426930    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.452650 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.452558    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:40.464091 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.458648    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj9xx\" (UniqueName: \"kubernetes.io/projected/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-kube-api-access-gj9xx\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.465104 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.464615    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.535585 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.535548    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:40.788860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.784009    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:40.872958 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.872925    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"]
Nov 03 19:40:40.874905 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.874232    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:40:40.875173 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.875142    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:40:40.875359 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.875334    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:40:40.880503 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:40.880474    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:41.326132 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:41.325024    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc" event={"ID":"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3","Type":"ContainerStarted","Data":"0a8ce2fb5a7ef6fbfb1dceb94852294f49490865af064c4cb1c6e59702abdaaa"}
Nov 03 19:40:41.326771 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:41.326744    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc" event={"ID":"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3","Type":"ContainerStarted","Data":"e07feca7b3aaaf69aa68a38c2a3e169fdfa6456c1d391fabecd975f11122a8a4"}
Nov 03 19:40:41.342887 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:41.333877    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj" event={"ID":"9fea1109-4466-401f-8de1-3c9fae530f70","Type":"ContainerStarted","Data":"26fd557315a82ab01b4184b1a866811260e9298aa88ea92b84d0cf9f5c926cee"}
Nov 03 19:40:41.342887 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:41.333955    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj" event={"ID":"9fea1109-4466-401f-8de1-3c9fae530f70","Type":"ContainerStarted","Data":"7b7caa6101499b58e18d0cc3d2bcb3d4792e8e3455c58758496547eb32e9acf1"}
Nov 03 19:40:41.355294 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:41.355232    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc" podStartSLOduration=1.355209694 podStartE2EDuration="1.355209694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:41.347953616 +0000 UTC m=+797.602260917" watchObservedRunningTime="2026-11-03 19:40:41.355209694 +0000 UTC m=+797.609516922"
Nov 03 19:40:41.527863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:41.527434    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:41.545862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:41.536890    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:41.806860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:41.791473    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 19:40:41.806860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:41.791540    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm" podUID="e6a4dcf8-18ac-443b-b78b-014a4e38ce34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 19:40:41.936856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:41.936187    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:41.943857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:41.942532    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 19:40:41.943857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:41.942591    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj" podUID="9fea1109-4466-401f-8de1-3c9fae530f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 19:40:42.504010 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:42.503817    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:42.537963 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:42.537595    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 19:40:42.537963 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:42.537656    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc" podUID="e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 19:40:42.804861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:42.802524    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 19:40:42.804861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:42.802591    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm" podUID="e6a4dcf8-18ac-443b-b78b-014a4e38ce34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 19:40:42.938374 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:42.936373    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 19:40:42.938374 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:42.936450    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj" podUID="9fea1109-4466-401f-8de1-3c9fae530f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 19:40:43.366279 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:40:43.365261    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:40:43.547525 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:43.547482    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 19:40:43.548056 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:43.547546    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc" podUID="e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 19:40:43.570101 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:43.570064    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:43.786412 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:43.786376    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 19:40:43.786701 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:43.786659    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm" podUID="e6a4dcf8-18ac-443b-b78b-014a4e38ce34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 19:40:43.924959 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:43.924920    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:44.013497 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:44.013459    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 19:40:44.017978 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:44.017940    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj" podUID="9fea1109-4466-401f-8de1-3c9fae530f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 19:40:44.542340 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:44.542298    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 19:40:44.542636 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:44.542601    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc" podUID="e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 19:40:44.690860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:44.690102    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:40:44.691339 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:44.691231    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:40:44.788341 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:44.788302    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 19:40:44.788622 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:44.788590    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm" podUID="e6a4dcf8-18ac-443b-b78b-014a4e38ce34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 19:40:44.939383 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:44.939290    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 19:40:44.939627 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:44.939599    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj" podUID="9fea1109-4466-401f-8de1-3c9fae530f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 19:40:45.079862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:45.078928    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:40:45.445862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:45.444257    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:40:45.518994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:45.518935    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:40:45.528990 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:45.528944    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:40:45.544305 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:45.544268    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 19:40:45.545084 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:45.545049    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc" podUID="e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 19:40:45.576958 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:45.576914    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:40:45.696913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:45.692924    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:40:45.696913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:45.693420    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:40:45.696913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:45.695201    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 19:40:45.696913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:45.695542    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:40:45.726859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:45.726761    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj" podStartSLOduration=6.726742672 podStartE2EDuration="6.726742672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:41.397349744 +0000 UTC m=+797.651656970" watchObservedRunningTime="2026-11-03 19:40:45.726742672 +0000 UTC m=+801.981049897"
Nov 03 19:40:45.787284 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:45.787246    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 19:40:45.787559 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:45.787527    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm" podUID="e6a4dcf8-18ac-443b-b78b-014a4e38ce34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 19:40:46.041916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:45.998005    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 19:40:46.041916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:45.998088    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj" podUID="9fea1109-4466-401f-8de1-3c9fae530f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 19:40:46.041916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.031076    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:46.176642 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.176611    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:40:46.356411 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.356298    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:40:46.471133 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.471092    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:40:46.490861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.472143    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:40:46.524039 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.523995    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:40:46.524408 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.524381    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:40:46.557861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.535947    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:40:46.557861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.536397    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:40:46.558913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.558882    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 19:40:46.559042 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.558940    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc" podUID="e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 19:40:46.640916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.618400    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:40:46.641610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.641581    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:40:46.700866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.696969    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 19:40:46.722929 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.722600    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:40:46.788665 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.788209    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 19:40:46.788665 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.788271    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm" podUID="e6a4dcf8-18ac-443b-b78b-014a4e38ce34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 19:40:46.911466 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.911062    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:40:46.941576 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.941081    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 19:40:46.941576 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:46.941148    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj" podUID="9fea1109-4466-401f-8de1-3c9fae530f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 19:40:47.160787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:47.159648    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:40:47.165792 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:47.165487    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:40:47.165792 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:47.165555    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:40:47.169854 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:47.167803    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:40:47.459337 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:47.459255    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:40:47.547859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:47.544552    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 19:40:47.547859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:47.544615    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc" podUID="e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 19:40:47.799921 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:47.795945    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 19:40:47.799921 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:47.799886    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm" podUID="e6a4dcf8-18ac-443b-b78b-014a4e38ce34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 19:40:47.933772 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:47.932396    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 19:40:47.933772 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:47.932468    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj" podUID="9fea1109-4466-401f-8de1-3c9fae530f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 19:40:48.016956 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:48.016919    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:48.536420 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:48.536377    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 19:40:48.536608 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:48.536440    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc" podUID="e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 19:40:49.038860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:49.037901    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:49.238260 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:49.238230    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:49.238940 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:49.238912    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:49.240994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:49.240973    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:40:49.274854 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:49.274805    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:49.275358 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:49.275336    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:49.277217 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:49.277196    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:40:49.536326 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:49.536289    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 19:40:49.536527 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:49.536356    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc" podUID="e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 19:40:49.907577 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:49.907481    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:50.541074 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:50.541043    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:50.541556 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:50.541338    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:50.542265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:50.542240    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:40:50.631893 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:50.631803    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:51.325135 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:51.325101    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:51.912852 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:51.912804    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:52.605846 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:52.605798    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:53.054326 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:53.054290    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:53.497151 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:53.497053    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:54.415013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:54.414978    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:55.242963 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:55.242924    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:56.116868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:56.116816    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:57.223042 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:57.223005    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:58.013374 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:58.013332    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:58.723946 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:58.723917    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:59.448503 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:40:59.448472    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:00.073984 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:00.073943    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:00.210858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:00.209881    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:00.729601 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:00.729569    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:01.418533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:01.418493    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:02.010815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:02.010779    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:02.942242 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:02.942198    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:03.619054 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:03.619024    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:04.361481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:04.361424    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:04.958944 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:04.958906    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:05.601901 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:05.601870    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:06.234402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.234366    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:06.702857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.702278    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"]
Nov 03 19:41:06.706566 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.706540    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.713622 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.713589    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"]
Nov 03 19:41:06.718193 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.718172    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2c3fc4\"/\"distinct-2-w1-2c3fc4-kserve-self-signed-certs\""
Nov 03 19:41:06.789928 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.789893    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-model-cache\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.790120 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.789955    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-dshm\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.790120 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.790047    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb69380-947e-474e-974f-9e17065379cf-tls-certs\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.790224 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.790169    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfwk5\" (UniqueName: \"kubernetes.io/projected/cfb69380-947e-474e-974f-9e17065379cf-kube-api-access-qfwk5\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.790293 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.790272    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-home\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.790352 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.790325    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-tmp-dir\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.866964 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.866923    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:06.898510 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.898464    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-home\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.898792 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.898771    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-tmp-dir\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.899070 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.899052    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-model-cache\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.899250 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.899232    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-dshm\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.899341 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.899329    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb69380-947e-474e-974f-9e17065379cf-tls-certs\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.899515 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.899499    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfwk5\" (UniqueName: \"kubernetes.io/projected/cfb69380-947e-474e-974f-9e17065379cf-kube-api-access-qfwk5\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.900452 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.900427    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-home\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.900869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.900830    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-tmp-dir\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.901299 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.901241    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-model-cache\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.912894 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.912859    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb69380-947e-474e-974f-9e17065379cf-tls-certs\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.913374 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.913310    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-dshm\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:06.928114 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:06.928077    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfwk5\" (UniqueName: \"kubernetes.io/projected/cfb69380-947e-474e-974f-9e17065379cf-kube-api-access-qfwk5\") pod \"distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") " pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:07.026741 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:07.026626    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:07.033123 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:07.033077    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:07.392136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:07.392109    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"]
Nov 03 19:41:07.394217 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:41:07.394189    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfb69380_947e_474e_974f_9e17065379cf.slice/crio-706a69ff82e9a8a3fff55ef6e0ce9c8a402d1aee69fb6464b6c7f26d5f07f5a0 WatchSource:0}: Error finding container 706a69ff82e9a8a3fff55ef6e0ce9c8a402d1aee69fb6464b6c7f26d5f07f5a0: Status 404 returned error can't find the container with id 706a69ff82e9a8a3fff55ef6e0ce9c8a402d1aee69fb6464b6c7f26d5f07f5a0
Nov 03 19:41:07.703936 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:07.703817    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:07.764373 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:07.764332    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" event={"ID":"cfb69380-947e-474e-974f-9e17065379cf","Type":"ContainerStarted","Data":"63a30f37d3853b3f6ea1ffecaf303f318e3302fb1743b9fccc7d587f57320424"}
Nov 03 19:41:07.764570 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:07.764384    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" event={"ID":"cfb69380-947e-474e-974f-9e17065379cf","Type":"ContainerStarted","Data":"706a69ff82e9a8a3fff55ef6e0ce9c8a402d1aee69fb6464b6c7f26d5f07f5a0"}
Nov 03 19:41:07.785768 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:07.785715    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podStartSLOduration=1.7857014310000001 podStartE2EDuration="1.785701431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:07.782934428 +0000 UTC m=+824.037241730" watchObservedRunningTime="2026-11-03 19:41:07.785701431 +0000 UTC m=+824.040008655"
Nov 03 19:41:08.034560 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:08.034532    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:08.409756 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:08.409673    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:09.079064 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:09.079025    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:09.723701 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:09.723648    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:09.935576 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:09.935528    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"]
Nov 03 19:41:09.939228 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:09.939211    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:09.953448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:09.953407    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-036184\"/\"distinct-2-w3-036184-kserve-self-signed-certs\""
Nov 03 19:41:09.965304 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:09.965275    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"]
Nov 03 19:41:10.025730 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.025647    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:10.067560 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.067519    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k2bh\" (UniqueName: \"kubernetes.io/projected/215c21e1-690b-4a4b-ae98-277e56811956-kube-api-access-2k2bh\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.067939 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.067915    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-tmp-dir\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.068237 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.068213    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-model-cache\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.068365 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.068264    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-dshm\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.068365 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.068306    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215c21e1-690b-4a4b-ae98-277e56811956-tls-certs\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.068365 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.068336    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-home\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.068545 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.068514    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"]
Nov 03 19:41:10.088118 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.088083    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"]
Nov 03 19:41:10.091977 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.088715    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.111976 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.111948    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ef8f84-openshift-default-dockercfg-77kt7\""
Nov 03 19:41:10.174219 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.174183    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-dshm\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.174500 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.174476    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.174632 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.174615    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215c21e1-690b-4a4b-ae98-277e56811956-tls-certs\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.174739 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.174726    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-home\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.174906 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.174889    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941d6f5f-47cd-439e-9463-ecc633799b05-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.175056 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.175040    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k2bh\" (UniqueName: \"kubernetes.io/projected/215c21e1-690b-4a4b-ae98-277e56811956-kube-api-access-2k2bh\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.175156 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.175142    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rph4z\" (UniqueName: \"kubernetes.io/projected/941d6f5f-47cd-439e-9463-ecc633799b05-kube-api-access-rph4z\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.175280 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.175266    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-tmp-dir\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.175445 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.175429    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.175639 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.175619    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.184411 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.182950    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941d6f5f-47cd-439e-9463-ecc633799b05-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.184411 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.183074    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.184411 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.183114    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.184411 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.183143    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/941d6f5f-47cd-439e-9463-ecc633799b05-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.184411 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.183188    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-model-cache\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.184411 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.183682    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-model-cache\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.184411 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.184270    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-tmp-dir\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.189511 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.188249    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-home\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.256217 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.242320    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215c21e1-690b-4a4b-ae98-277e56811956-tls-certs\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.256217 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.251122    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-dshm\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.256217 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.252250    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k2bh\" (UniqueName: \"kubernetes.io/projected/215c21e1-690b-4a4b-ae98-277e56811956-kube-api-access-2k2bh\") pod \"distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") " pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.256217 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.255531    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:10.285754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.284074    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.285754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.284115    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.285754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.284132    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/941d6f5f-47cd-439e-9463-ecc633799b05-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.285754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.284166    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.285754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.284206    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941d6f5f-47cd-439e-9463-ecc633799b05-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.285754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.284239    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rph4z\" (UniqueName: \"kubernetes.io/projected/941d6f5f-47cd-439e-9463-ecc633799b05-kube-api-access-rph4z\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.285754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.284274    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.285754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.284308    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.285754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.284328    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941d6f5f-47cd-439e-9463-ecc633799b05-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.285754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.284960    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.285754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.285129    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.285754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.285513    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.285754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.285681    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.286532 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.285933    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941d6f5f-47cd-439e-9463-ecc633799b05-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.287791 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.287518    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.288951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.288469    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941d6f5f-47cd-439e-9463-ecc633799b05-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.298006 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.297706    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rph4z\" (UniqueName: \"kubernetes.io/projected/941d6f5f-47cd-439e-9463-ecc633799b05-kube-api-access-rph4z\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.299590 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.299558    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/941d6f5f-47cd-439e-9463-ecc633799b05-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.430795 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.430762    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:10.555035 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.555007    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"]
Nov 03 19:41:10.556907 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:41:10.556878    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod215c21e1_690b_4a4b_ae98_277e56811956.slice/crio-f6b4c9dcc15e5abf60441ce0009181f65b0f6e8e899b7e03feed4e367b23da38 WatchSource:0}: Error finding container f6b4c9dcc15e5abf60441ce0009181f65b0f6e8e899b7e03feed4e367b23da38: Status 404 returned error can't find the container with id f6b4c9dcc15e5abf60441ce0009181f65b0f6e8e899b7e03feed4e367b23da38
Nov 03 19:41:10.646382 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.646337    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:10.728093 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.728060    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"]
Nov 03 19:41:10.730153 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:41:10.730123    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941d6f5f_47cd_439e_9463_ecc633799b05.slice/crio-4d18194edd45fa78249e4a4f48ffd7f43b23e24a8ce604b4e1b4ae6ec07816fb WatchSource:0}: Error finding container 4d18194edd45fa78249e4a4f48ffd7f43b23e24a8ce604b4e1b4ae6ec07816fb: Status 404 returned error can't find the container with id 4d18194edd45fa78249e4a4f48ffd7f43b23e24a8ce604b4e1b4ae6ec07816fb
Nov 03 19:41:10.732784 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.732747    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:10.732909 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.732815    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:10.732909 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.732876    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:10.799230 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.799189    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q" event={"ID":"941d6f5f-47cd-439e-9463-ecc633799b05","Type":"ContainerStarted","Data":"799963e23028ff1be61f73a5f7e79717bc8f8a6f9eb83ff9c19d2ae59aa5cdd4"}
Nov 03 19:41:10.799346 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.799250    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q" event={"ID":"941d6f5f-47cd-439e-9463-ecc633799b05","Type":"ContainerStarted","Data":"4d18194edd45fa78249e4a4f48ffd7f43b23e24a8ce604b4e1b4ae6ec07816fb"}
Nov 03 19:41:10.801258 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.801234    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" event={"ID":"215c21e1-690b-4a4b-ae98-277e56811956","Type":"ContainerStarted","Data":"db72d8906736ea92a286407b8bcc6306bfebc46a6ed33f27f479f8e84d659399"}
Nov 03 19:41:10.801346 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.801271    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" event={"ID":"215c21e1-690b-4a4b-ae98-277e56811956","Type":"ContainerStarted","Data":"f6b4c9dcc15e5abf60441ce0009181f65b0f6e8e899b7e03feed4e367b23da38"}
Nov 03 19:41:10.830619 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:10.830508    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q" podStartSLOduration=0.830487431 podStartE2EDuration="830.487431ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:10.827728922 +0000 UTC m=+827.082036147" watchObservedRunningTime="2026-11-03 19:41:10.830487431 +0000 UTC m=+827.084794656"
Nov 03 19:41:11.259949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:11.257981    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:11.431091 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:11.431059    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:11.432712 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:11.432682    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 19:41:11.432862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:11.432735    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q" podUID="941d6f5f-47cd-439e-9463-ecc633799b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 19:41:11.539783 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:11.539742    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:12.432367 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:12.432328    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 19:41:12.432952 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:12.432915    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q" podUID="941d6f5f-47cd-439e-9463-ecc633799b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 19:41:13.279247 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:41:13.279102    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:41:13.302427 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:13.302370    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:13.431276 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:13.431240    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 19:41:13.431488 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:13.431297    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q" podUID="941d6f5f-47cd-439e-9463-ecc633799b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 19:41:13.545416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:13.545377    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:14.341598 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:14.341552    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:14.433858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:14.432066    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 19:41:14.433858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:14.432137    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q" podUID="941d6f5f-47cd-439e-9463-ecc633799b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 19:41:15.077401 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.077363    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:15.313726 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.313658    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podStartSLOduration=6.313639089 podStartE2EDuration="6.313639089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:10.856947067 +0000 UTC m=+827.111254293" watchObservedRunningTime="2026-11-03 19:41:15.313639089 +0000 UTC m=+831.567946315"
Nov 03 19:41:15.315398 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.315265    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"]
Nov 03 19:41:15.321509 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.321488    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.326303 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.326278    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"]
Nov 03 19:41:15.401860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.385684    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:15.409710 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.409681    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-05e217\"/\"distinct-2-w5-05e217-kserve-self-signed-certs\""
Nov 03 19:41:15.415332 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.415292    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-tls-certs\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.415548 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.415536    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-dshm\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.415666 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.415653    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66ph5\" (UniqueName: \"kubernetes.io/projected/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-kube-api-access-66ph5\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.415810 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.415798    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-model-cache\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.415953 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.415933    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-home\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.416028 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.416015    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-tmp-dir\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.431829 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.431397    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 19:41:15.431829 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.431461    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q" podUID="941d6f5f-47cd-439e-9463-ecc633799b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 19:41:15.519979 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.519940    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-tls-certs\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.520184 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.520000    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-dshm\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.520184 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.520031    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66ph5\" (UniqueName: \"kubernetes.io/projected/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-kube-api-access-66ph5\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.520184 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.520098    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-model-cache\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.520184 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.520141    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-home\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.520400 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.520229    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-tmp-dir\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.520773 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.520743    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-tmp-dir\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.521919 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.521738    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-home\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.522188 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.522139    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-model-cache\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.535202 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.534606    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-dshm\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.535202 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.535152    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-tls-certs\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.557865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.556851    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66ph5\" (UniqueName: \"kubernetes.io/projected/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-kube-api-access-66ph5\") pod \"distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") " pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:15.640915 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:15.640065    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:16.097615 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:16.097584    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"]
Nov 03 19:41:16.098655 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:41:16.098607    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdf93f15_df9e_4a66_8d89_4d412d3bf35c.slice/crio-4244b6f50ce3840f52556f9e25f4261957078c916172777ca6ea9e441f7fd9ed WatchSource:0}: Error finding container 4244b6f50ce3840f52556f9e25f4261957078c916172777ca6ea9e441f7fd9ed: Status 404 returned error can't find the container with id 4244b6f50ce3840f52556f9e25f4261957078c916172777ca6ea9e441f7fd9ed
Nov 03 19:41:16.122295 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:16.122260    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:16.431985 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:16.431900    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 19:41:16.431985 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:16.431959    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q" podUID="941d6f5f-47cd-439e-9463-ecc633799b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 19:41:16.867776 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:16.867739    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:16.887514 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:16.885183    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" event={"ID":"bdf93f15-df9e-4a66-8d89-4d412d3bf35c","Type":"ContainerStarted","Data":"86a4b06b644d62963636ff81db722847f27161ff9426e6fdea440dbfdb5b3bbc"}
Nov 03 19:41:16.887514 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:16.885240    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" event={"ID":"bdf93f15-df9e-4a66-8d89-4d412d3bf35c","Type":"ContainerStarted","Data":"4244b6f50ce3840f52556f9e25f4261957078c916172777ca6ea9e441f7fd9ed"}
Nov 03 19:41:16.924557 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:16.920657    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podStartSLOduration=1.920634729 podStartE2EDuration="1.920634729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:16.917090047 +0000 UTC m=+833.171397273" watchObservedRunningTime="2026-11-03 19:41:16.920634729 +0000 UTC m=+833.174941955"
Nov 03 19:41:17.040857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:17.039928    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:17.101090 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:17.101058    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:17.431693 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:17.431641    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 19:41:17.431892 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:17.431721    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q" podUID="941d6f5f-47cd-439e-9463-ecc633799b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 19:41:17.533441 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:17.533403    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:17.644869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:17.641134    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:17.761373 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:17.761278    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:17.948156 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:17.948126    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:18.446560 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:18.446523    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:18.708487 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:18.708406    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:18.709759 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:18.709733    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:18.711915 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:18.711888    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:41:19.136659 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:19.136620    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:19.741892 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:19.741853    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:19.958702 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:19.957384    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:20.256694 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:20.256661    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:20.269510 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:20.269486    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:20.864863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:20.862159    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:20.981490 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:20.981458    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:21.704860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:21.700776    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:22.358935 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:22.356892    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:23.033416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:23.033013    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:23.639882 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:23.639845    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:24.404439 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.404399    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:24.570785 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.570746    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"]
Nov 03 19:41:24.576514 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.576484    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.593876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.593232    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"]
Nov 03 19:41:24.631227 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.627900    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"]
Nov 03 19:41:24.632239 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.632213    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.661455 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.658947    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"]
Nov 03 19:41:24.707481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.707440    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4vcz\" (UniqueName: \"kubernetes.io/projected/1c377e40-757e-47a6-b015-bf914f369081-kube-api-access-x4vcz\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.707642 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.707486    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.707642 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.707518    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c377e40-757e-47a6-b015-bf914f369081-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.707752 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.707723    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.707886 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.707824    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c377e40-757e-47a6-b015-bf914f369081-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.707950 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.707928    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.708028 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.708013    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c377e40-757e-47a6-b015-bf914f369081-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.708065 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.708043    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.708065 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.708060    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.774868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.774801    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"]
Nov 03 19:41:24.779399 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.779378    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:24.783675 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.783644    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-435bd8-openshift-default-dockercfg-zznl2\""
Nov 03 19:41:24.792084 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.792060    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"]
Nov 03 19:41:24.809145 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809110    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.809318 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809178    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c377e40-757e-47a6-b015-bf914f369081-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.809318 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809214    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.809318 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809237    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.809318 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809287    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.809531 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809323    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.809531 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809387    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4vcz\" (UniqueName: \"kubernetes.io/projected/1c377e40-757e-47a6-b015-bf914f369081-kube-api-access-x4vcz\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.809531 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809423    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.809531 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809446    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c377e40-757e-47a6-b015-bf914f369081-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.809531 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809487    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.809788 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809532    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.809788 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809576    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.809788 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809603    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.809788 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809605    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.809788 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809685    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.809788 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809713    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5nwc\" (UniqueName: \"kubernetes.io/projected/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-kube-api-access-g5nwc\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.809788 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809743    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.810190 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809815    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c377e40-757e-47a6-b015-bf914f369081-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.810190 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.809864    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.810318 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.810298    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.810436 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.810412    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.810708 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.810686    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.811159 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.811134    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c377e40-757e-47a6-b015-bf914f369081-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.812964 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.812935    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.813348 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.813327    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c377e40-757e-47a6-b015-bf914f369081-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.834436 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.829458    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4vcz\" (UniqueName: \"kubernetes.io/projected/1c377e40-757e-47a6-b015-bf914f369081-kube-api-access-x4vcz\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.843507 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.843476    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c377e40-757e-47a6-b015-bf914f369081-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.894658 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.894617    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:24.911714 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.911564    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.911714 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.911640    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.911714 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.911674    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:24.912004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.911753    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:24.912004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.911868    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.912004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.911912    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:24.912004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.911953    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.912004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.911989    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:24.912270 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.912020    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.912270 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.912021    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.912270 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.912046    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.912270 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.912102    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.912270 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.912126    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5nwc\" (UniqueName: \"kubernetes.io/projected/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-kube-api-access-g5nwc\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.912270 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.912152    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:24.912270 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.912176    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ggjh\" (UniqueName: \"kubernetes.io/projected/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-kube-api-access-4ggjh\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:24.912270 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.912216    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:24.912675 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.912278    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.912675 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.912302    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.912675 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.912306    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:24.912675 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.912602    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.912906 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.912706    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.913068 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.912965    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.913316 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.913245    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:24.915044 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.915013    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.916025 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.915631    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.927985 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.927731    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.927985 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.927924    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5nwc\" (UniqueName: \"kubernetes.io/projected/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-kube-api-access-g5nwc\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:24.959800 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:24.959763    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:25.014221 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.014150    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.014221 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.014223    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.014472 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.014249    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.014472 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.014277    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.014472 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.014320    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.014472 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.014337    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ggjh\" (UniqueName: \"kubernetes.io/projected/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-kube-api-access-4ggjh\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.014472 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.014361    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.014472 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.014404    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.014472 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.014453    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.015589 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.015557    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.015890 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.015865    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.016000 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.015894    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.016128 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.016110    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.016187 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.016166    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.018867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.018080    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.019798 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.019777    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.030120 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.030088    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.030759 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.030737    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ggjh\" (UniqueName: \"kubernetes.io/projected/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-kube-api-access-4ggjh\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.095751 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.095709    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:25.200878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.200829    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"]
Nov 03 19:41:25.211440 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.204299    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:25.211440 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.204384    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:25.211440 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.204427    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:25.257045 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.255128    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:25.314180 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.310465    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"]
Nov 03 19:41:25.332879 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:41:25.329373    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad53de78_95ea_4f6b_b6a2_caa3e86bd49b.slice/crio-35b540413ed684c047894a18c7c08566caa48661460c5aec43f135470f34c672 WatchSource:0}: Error finding container 35b540413ed684c047894a18c7c08566caa48661460c5aec43f135470f34c672: Status 404 returned error can't find the container with id 35b540413ed684c047894a18c7c08566caa48661460c5aec43f135470f34c672
Nov 03 19:41:25.347183 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.344215    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:25.347183 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.344303    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:25.347183 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.344348    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:25.397209 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.397172    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"]
Nov 03 19:41:25.406239 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.406197    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:25.406329 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.406297    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:25.406384 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.406348    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:25.641553 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.640690    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:25.661877 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:25.660637    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:26.001742 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.001710    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk" event={"ID":"1c377e40-757e-47a6-b015-bf914f369081","Type":"ContainerStarted","Data":"56934bdc39df6cff78c53eba2f07cb66fe21aeebde40e4bc4a4ed245a5c607e7"}
Nov 03 19:41:26.001959 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.001772    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk" event={"ID":"1c377e40-757e-47a6-b015-bf914f369081","Type":"ContainerStarted","Data":"5d9e0c734b7f6d3da0e1b6f4ad871a424bfb765bdac36a04b938a1288a246b26"}
Nov 03 19:41:26.003708 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.003674    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q" event={"ID":"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b","Type":"ContainerStarted","Data":"9befdf34124b75d112ad5fd1b1501c36aa97a372c830cd4ffa78eedbb9acb680"}
Nov 03 19:41:26.003886 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.003729    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q" event={"ID":"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b","Type":"ContainerStarted","Data":"35b540413ed684c047894a18c7c08566caa48661460c5aec43f135470f34c672"}
Nov 03 19:41:26.005452 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.005420    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" event={"ID":"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f","Type":"ContainerStarted","Data":"a1e7d01aece73653fc1cd8e5ffee735994481db53489c9378869bfba2d50553e"}
Nov 03 19:41:26.005596 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.005475    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" event={"ID":"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f","Type":"ContainerStarted","Data":"245b1733a50762672fc16d85e80287d0186d562c42af19173cb1294199e0beb4"}
Nov 03 19:41:26.018012 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.017982    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:26.026419 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.026359    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk" podStartSLOduration=2.026342807 podStartE2EDuration="2.026342807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:26.024222587 +0000 UTC m=+842.278529813" watchObservedRunningTime="2026-11-03 19:41:26.026342807 +0000 UTC m=+842.280650032"
Nov 03 19:41:26.048651 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.048577    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q" podStartSLOduration=2.048555781 podStartE2EDuration="2.048555781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:26.046299716 +0000 UTC m=+842.300606943" watchObservedRunningTime="2026-11-03 19:41:26.048555781 +0000 UTC m=+842.302863008"
Nov 03 19:41:26.100112 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.097165    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:26.107243 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.103008    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 19:41:26.107243 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.103106    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" podUID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 19:41:26.115860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.112540    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" podStartSLOduration=2.1125183180000002 podStartE2EDuration="2.112518318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:26.084197972 +0000 UTC m=+842.338505201" watchObservedRunningTime="2026-11-03 19:41:26.112518318 +0000 UTC m=+842.366825543"
Nov 03 19:41:26.118422 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.118387    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:26.832858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.831928    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"]
Nov 03 19:41:26.864881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.860716    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:26.899900 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.890128    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"]
Nov 03 19:41:26.899900 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.890318    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:26.944860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.914892    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:26.948502 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.947238    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c5f6cb\"/\"distinct-2-w6-c5f6cb-kserve-self-signed-certs\""
Nov 03 19:41:26.948502 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.947368    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c5f6cb\"/\"default-dockercfg-9bbw9\""
Nov 03 19:41:26.948502 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.947451    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-c5f6cb\"/\"kube-root-ca.crt\""
Nov 03 19:41:26.948502 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.947533    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-c5f6cb\"/\"openshift-service-ca.crt\""
Nov 03 19:41:27.095027 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:26.974262    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:27.095027 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.006389    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-model-cache\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.095027 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.006450    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-tmp-dir\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.095027 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.006521    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-dshm\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.095027 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.006594    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj2bc\" (UniqueName: \"kubernetes.io/projected/04d00e3f-7b7f-4bde-a443-77aadc84642b-kube-api-access-zj2bc\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.095027 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.006669    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-home\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.095027 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.006765    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04d00e3f-7b7f-4bde-a443-77aadc84642b-tls-certs\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.095027 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.007469    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:41:27.095027 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.007522    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk" podUID="1c377e40-757e-47a6-b015-bf914f369081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:41:27.097279 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.096151    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 19:41:27.097279 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.096229    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" podUID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 19:41:27.127677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.107896    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj2bc\" (UniqueName: \"kubernetes.io/projected/04d00e3f-7b7f-4bde-a443-77aadc84642b-kube-api-access-zj2bc\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.127677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.107958    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-home\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.127677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.108060    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04d00e3f-7b7f-4bde-a443-77aadc84642b-tls-certs\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.127677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.108166    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-model-cache\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.127677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.108212    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-tmp-dir\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.127677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.108288    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-dshm\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.127677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.109374    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-home\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.127677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.109661    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-model-cache\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.127677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.109981    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-tmp-dir\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.164714 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.162081    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:41:27.164714 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.162142    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q" podUID="ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:41:27.401380 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.373293    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04d00e3f-7b7f-4bde-a443-77aadc84642b-tls-certs\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.401380 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.374158    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj2bc\" (UniqueName: \"kubernetes.io/projected/04d00e3f-7b7f-4bde-a443-77aadc84642b-kube-api-access-zj2bc\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.401380 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.383681    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-dshm\") pod \"distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.471576 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.471535    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:27.965901 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.965214    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:41:27.965901 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:27.965274    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk" podUID="1c377e40-757e-47a6-b015-bf914f369081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:41:28.063926 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:28.063641    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:41:28.063926 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:28.063708    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q" podUID="ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:41:28.064912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:28.064881    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:28.097575 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:28.097381    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 19:41:28.097575 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:28.097447    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" podUID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 19:41:28.377895 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:28.364891    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"]
Nov 03 19:41:28.895858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:28.895794    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:41:28.896056 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:28.895876    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk" podUID="1c377e40-757e-47a6-b015-bf914f369081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:41:28.963083 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:28.961624    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:41:28.963083 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:28.961686    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q" podUID="ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:41:29.129881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:29.119761    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 19:41:29.129881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:29.119847    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" podUID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 19:41:29.167479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:29.164310    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:29.402051 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:29.402015    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" event={"ID":"04d00e3f-7b7f-4bde-a443-77aadc84642b","Type":"ContainerStarted","Data":"a5293711efcc818e407dd8706050ed530915acf3e02106eb393fbb12bf47b021"}
Nov 03 19:41:29.403280 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:29.403255    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" event={"ID":"04d00e3f-7b7f-4bde-a443-77aadc84642b","Type":"ContainerStarted","Data":"f78ba6e360af28f53fe4863edfb57c35a252a6cbb57a8b7608b068f11e80096c"}
Nov 03 19:41:29.435005 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:29.434909    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podStartSLOduration=3.434886969 podStartE2EDuration="3.434886969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:29.433288627 +0000 UTC m=+845.687595848" watchObservedRunningTime="2026-11-03 19:41:29.434886969 +0000 UTC m=+845.689194194"
Nov 03 19:41:29.476863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:29.474976    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:29.738020 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:29.737906    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:29.907299 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:29.907260    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:41:29.907554 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:29.907525    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk" podUID="1c377e40-757e-47a6-b015-bf914f369081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:41:29.966859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:29.966293    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:41:29.966859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:29.966382    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q" podUID="ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:41:30.098276 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:30.098237    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 19:41:30.129874 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:30.109978    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" podUID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 19:41:30.190875 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:30.179748    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:30.274989 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:30.274934    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:30.299910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:30.288371    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:30.397930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:30.390750    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:30.468111 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:30.468063    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:30.486932 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:30.481632    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:30.724962 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:30.719412    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded"
Nov 03 19:41:30.898322 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:30.898281    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:41:30.899546 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:30.899509    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk" podUID="1c377e40-757e-47a6-b015-bf914f369081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:41:30.962286 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:30.962250    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:41:30.962529 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:30.962500    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q" podUID="ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:41:30.984268 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:30.984158    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:31.015994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:31.015946    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:31.098787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:31.098326    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 19:41:31.098787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:31.098390    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" podUID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 19:41:31.181099 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:31.181031    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:31.182861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:31.182807    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:31.237112 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:31.236985    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 19:41:31.239161 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:31.239137    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:41:31.281992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:31.281945    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:31.285009 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:31.284981    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:41:31.464287 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:31.464239    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:41:31.484864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:31.464800    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:41:31.485429 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:31.485391    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 19:41:31.486000 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:31.485979    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:41:31.597963 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:31.534455    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:31.597963 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:31.534908    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:41:31.979273 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:31.979167    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:41:31.979541 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:31.979508    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk" podUID="1c377e40-757e-47a6-b015-bf914f369081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:41:32.019039 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.018989    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:32.041928 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.041890    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:41:32.042207 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.042178    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q" podUID="ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:41:32.075866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.073103    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:41:32.118860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.104323    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:32.118860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.109573    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 19:41:32.118860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.109644    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" podUID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 19:41:32.118860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.114524    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:32.183148 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.183094    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 19:41:32.185090 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.185042    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 19:41:32.289935 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.193768    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:32.343864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.240528    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:32.401909 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.288884    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:41:32.401909 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.378972    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:32.401909 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.379073    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:32.401909 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.379140    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:41:32.401909 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.379196    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:41:32.532669 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.532607    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:32.536141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.536097    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:32.538576 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.538520    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:32.557743 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.557657    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:41:32.939089 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.938997    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:41:32.939344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.939314    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk" podUID="1c377e40-757e-47a6-b015-bf914f369081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:41:32.997236 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:32.997178    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:41:33.013971 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.013898    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q" podUID="ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:41:33.020379 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.020336    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:33.091763 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.087184    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:33.091763 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.087797    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:33.110946 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.110895    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 19:41:33.114993 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.114935    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" podUID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 19:41:33.118949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.118907    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:33.248060 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.247959    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:33.324045 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.323996    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:33.378111 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.378049    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 19:41:33.386189 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.378868    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:33.543662 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.543035    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:33.551864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.548598    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 19:41:33.566862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.552893    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:33.778043 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.777999    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:41:33.974928 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.967892    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:41:33.974928 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.967975    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk" podUID="1c377e40-757e-47a6-b015-bf914f369081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:41:33.987881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.985617    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:41:33.987881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:33.985676    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q" podUID="ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:41:34.041050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.028152    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:34.135132 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.116940    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 19:41:34.135132 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.116996    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" podUID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 19:41:34.135132 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.128507    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:34.189779 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.186186    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 19:41:34.393884 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.369274    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:41:34.395263 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.395221    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:34.396494 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.396461    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:41:34.449597 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.449558    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:34.544402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.544285    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:41:34.562799 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.562745    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 19:41:34.589153 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.589101    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 19:41:34.772912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.772808    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:41:34.859031 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.858209    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:41:34.868853 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.860272    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:41:34.868853 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.862356    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:41:34.881203 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.869245    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:41:34.927861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.897964    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:41:34.927861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.898022    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk" podUID="1c377e40-757e-47a6-b015-bf914f369081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:41:34.942641 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.942608    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:41:34.944120 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.944101    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:41:34.951372 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.951347    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:41:34.960948 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.960918    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:41:34.961123 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:34.960972    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q" podUID="ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:41:35.097479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:35.097443    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 19:41:35.097817 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:35.097786    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" podUID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 19:41:35.737569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:35.737528    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:36.538757 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:36.538725    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:36.540194 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:36.540150    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:36.564208 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:36.564172    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:41:36.602856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:36.597974    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:36.603680 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:36.603558    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:36.605539 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:36.605485    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:41:36.730862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:36.728690    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:36.943132 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:36.943031    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:41:36.943132 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:36.943107    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk" podUID="1c377e40-757e-47a6-b015-bf914f369081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:41:37.476939 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:37.476908    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:37.502400 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:37.502371    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:37.603153 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:37.603119    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:37.604903 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:37.604880    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:37.605959 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:37.605934    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:41:37.886351 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:37.886308    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:38.265906 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:38.265807    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:41:38.747216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:38.747176    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:39.573551 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:39.573459    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:40.217215 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.217178    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"]
Nov 03 19:41:40.221170 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.221150    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.247850 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.247810    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"]
Nov 03 19:41:40.309750 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.309720    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.310000 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.309776    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.310079 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.309994    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.310079 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.310033    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.310179 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.310121    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.310179 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.310144    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98r7z\" (UniqueName: \"kubernetes.io/projected/e56d7264-7d35-4dd8-946b-c1ab4d40e941-kube-api-access-98r7z\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.310290 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.310212    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.310290 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.310265    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.310358 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.310309    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.420809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.417941    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.420809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.418032    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.420809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.418178    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.420809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.418209    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.420809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.418263    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.420809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.418290    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98r7z\" (UniqueName: \"kubernetes.io/projected/e56d7264-7d35-4dd8-946b-c1ab4d40e941-kube-api-access-98r7z\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.420809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.418317    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.420809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.418379    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.420809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.418405    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.420809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.419232    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.420809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.419519    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.420809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.419769    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.420809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.420418    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.432858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.429441    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.434168 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.433199    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.486716 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.438825    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.486716 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.454609    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.522799 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.508734    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"]
Nov 03 19:41:40.555992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.555214    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98r7z\" (UniqueName: \"kubernetes.io/projected/e56d7264-7d35-4dd8-946b-c1ab4d40e941-kube-api-access-98r7z\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.559025 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.557642    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:40.560513 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.559966    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.567122 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.566990    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"]
Nov 03 19:41:40.604344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.604311    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"]
Nov 03 19:41:40.608261 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.608240    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.653192 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.653158    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"]
Nov 03 19:41:40.736687 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.736641    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84vtw\" (UniqueName: \"kubernetes.io/projected/7ca1424b-adf4-42f6-8165-44af258b508c-kube-api-access-84vtw\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.736911 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.736695    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.736911 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.736872    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.736911 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.736905    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.737042 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.736933    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca1424b-adf4-42f6-8165-44af258b508c-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.737078 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.737035    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.737111 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.737086    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca1424b-adf4-42f6-8165-44af258b508c-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.737148 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.737109    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.737189 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.737173    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff03678-8a48-4bdb-83c5-4941236f45bc-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.737256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.737244    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca1424b-adf4-42f6-8165-44af258b508c-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.737290 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.737272    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.737325 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.737299    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.737325 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.737318    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n625j\" (UniqueName: \"kubernetes.io/projected/6ff03678-8a48-4bdb-83c5-4941236f45bc-kube-api-access-n625j\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.737447 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.737435    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.737494 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.737462    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.737528 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.737495    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.737582 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.737570    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.737617 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.737590    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.834294 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.834260    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:40.838963 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.838931    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84vtw\" (UniqueName: \"kubernetes.io/projected/7ca1424b-adf4-42f6-8165-44af258b508c-kube-api-access-84vtw\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.839109 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.838976    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.839109 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.839041    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.839109 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.839069    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.839109 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.839102    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca1424b-adf4-42f6-8165-44af258b508c-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.839330 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.839163    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.839330 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.839225    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca1424b-adf4-42f6-8165-44af258b508c-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.839330 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.839247    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.839330 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.839282    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff03678-8a48-4bdb-83c5-4941236f45bc-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.839541 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.839443    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.839608 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.839591    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.839664 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.839642    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.839806 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.839777    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.840037 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.840015    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca1424b-adf4-42f6-8165-44af258b508c-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.840247 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.840224    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca1424b-adf4-42f6-8165-44af258b508c-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.840344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.840273    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.840344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.840308    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.840344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.840335    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n625j\" (UniqueName: \"kubernetes.io/projected/6ff03678-8a48-4bdb-83c5-4941236f45bc-kube-api-access-n625j\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.840567 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.840402    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.840567 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.840435    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.840691 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.840664    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.840748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.840712    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff03678-8a48-4bdb-83c5-4941236f45bc-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.840748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.840724    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.840879 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.840740    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.840879 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.840749    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.841015 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.840998    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.841215 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.841193    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.841433 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.841411    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.841997 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.841972    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.842717 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.842694    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca1424b-adf4-42f6-8165-44af258b508c-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.843020 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.842994    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.843190 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.843175    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.865025 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.864994    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.865166 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.865040    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84vtw\" (UniqueName: \"kubernetes.io/projected/7ca1424b-adf4-42f6-8165-44af258b508c-kube-api-access-84vtw\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.865682 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.865656    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca1424b-adf4-42f6-8165-44af258b508c-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:40.865771 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.865737    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n625j\" (UniqueName: \"kubernetes.io/projected/6ff03678-8a48-4bdb-83c5-4941236f45bc-kube-api-access-n625j\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.889518 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.889484    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:40.923239 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:40.923204    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:41.131140 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.131110    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"]
Nov 03 19:41:41.132392 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:41:41.132357    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56d7264_7d35_4dd8_946b_c1ab4d40e941.slice/crio-3f3da7d2352f771d57a5bcb80cfe67f74a6bbdd8a05353d226e9c2ba3529b8ed WatchSource:0}: Error finding container 3f3da7d2352f771d57a5bcb80cfe67f74a6bbdd8a05353d226e9c2ba3529b8ed: Status 404 returned error can't find the container with id 3f3da7d2352f771d57a5bcb80cfe67f74a6bbdd8a05353d226e9c2ba3529b8ed
Nov 03 19:41:41.135193 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.135161    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:41.135257 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.135237    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:41.135298 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.135275    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:41.211517 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.211488    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"]
Nov 03 19:41:41.215846 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:41:41.215809    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff03678_8a48_4bdb_83c5_4941236f45bc.slice/crio-e1b94f36c08b876818e614a42543109cd41bb633b4095345abd057ee0e65e486 WatchSource:0}: Error finding container e1b94f36c08b876818e614a42543109cd41bb633b4095345abd057ee0e65e486: Status 404 returned error can't find the container with id e1b94f36c08b876818e614a42543109cd41bb633b4095345abd057ee0e65e486
Nov 03 19:41:41.223221 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.223193    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"]
Nov 03 19:41:41.224914 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.224872    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:41.225019 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.224963    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:41.225019 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.225007    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:41.227372 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:41:41.227344    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca1424b_adf4_42f6_8165_44af258b508c.slice/crio-1af3400e649de1402548e2accdeca79e7d2af0b23a370961bd52635da1df3f54 WatchSource:0}: Error finding container 1af3400e649de1402548e2accdeca79e7d2af0b23a370961bd52635da1df3f54: Status 404 returned error can't find the container with id 1af3400e649de1402548e2accdeca79e7d2af0b23a370961bd52635da1df3f54
Nov 03 19:41:41.234861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.234813    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:41.234978 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.234896    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:41.234978 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.234926    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:41.278614 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.278576    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp" event={"ID":"6ff03678-8a48-4bdb-83c5-4941236f45bc","Type":"ContainerStarted","Data":"e1b94f36c08b876818e614a42543109cd41bb633b4095345abd057ee0e65e486"}
Nov 03 19:41:41.281667 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.281568    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn" event={"ID":"e56d7264-7d35-4dd8-946b-c1ab4d40e941","Type":"ContainerStarted","Data":"ede69654eeddbd2f71333aacdeaeac2655cd04ffe0046ba709d74a004459b88f"}
Nov 03 19:41:41.281667 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.281619    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn" event={"ID":"e56d7264-7d35-4dd8-946b-c1ab4d40e941","Type":"ContainerStarted","Data":"3f3da7d2352f771d57a5bcb80cfe67f74a6bbdd8a05353d226e9c2ba3529b8ed"}
Nov 03 19:41:41.284164 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.284097    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9" event={"ID":"7ca1424b-adf4-42f6-8165-44af258b508c","Type":"ContainerStarted","Data":"1af3400e649de1402548e2accdeca79e7d2af0b23a370961bd52635da1df3f54"}
Nov 03 19:41:41.310509 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.310436    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn" podStartSLOduration=1.310417291 podStartE2EDuration="1.310417291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:41.308119321 +0000 UTC m=+857.562426555" watchObservedRunningTime="2026-11-03 19:41:41.310417291 +0000 UTC m=+857.564724517"
Nov 03 19:41:41.548884 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.548848    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:41.834819 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:41.834742    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:42.350135 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:42.350088    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp" event={"ID":"6ff03678-8a48-4bdb-83c5-4941236f45bc","Type":"ContainerStarted","Data":"11fa0d117b7eb1bfd38ec5cb6e216ace2d7d87e5c89c67d0400546bb1ea1c78a"}
Nov 03 19:41:42.356100 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:42.356045    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9" event={"ID":"7ca1424b-adf4-42f6-8165-44af258b508c","Type":"ContainerStarted","Data":"863cc691deb31ce822c416bf6f606e67b0889175f0f4e6ea7c48f6039d709292"}
Nov 03 19:41:42.382858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:42.382136    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp" podStartSLOduration=2.382115716 podStartE2EDuration="2.382115716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:42.381219722 +0000 UTC m=+858.635526947" watchObservedRunningTime="2026-11-03 19:41:42.382115716 +0000 UTC m=+858.636422942"
Nov 03 19:41:42.421001 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:42.420937    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9" podStartSLOduration=2.420916536 podStartE2EDuration="2.420916536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:42.420415239 +0000 UTC m=+858.674722466" watchObservedRunningTime="2026-11-03 19:41:42.420916536 +0000 UTC m=+858.675223766"
Nov 03 19:41:42.444795 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:42.444755    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:42.850992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:42.850949    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 19:41:42.851516 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:42.851002    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn" podUID="e56d7264-7d35-4dd8-946b-c1ab4d40e941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 19:41:42.890064 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:42.890028    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:42.892056 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:42.891966    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:41:42.892280 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:42.892258    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp" podUID="6ff03678-8a48-4bdb-83c5-4941236f45bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 19:41:42.928872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:42.927212    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:42.933864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:42.930942    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 19:41:42.933864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:42.931000    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9" podUID="7ca1424b-adf4-42f6-8165-44af258b508c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 19:41:43.361720 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:41:43.361676    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:41:43.450520 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:43.449144    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:43.839856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:43.839259    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 19:41:43.839856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:43.839323    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn" podUID="e56d7264-7d35-4dd8-946b-c1ab4d40e941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 19:41:43.894296 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:43.894256    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:41:43.894913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:43.894878    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp" podUID="6ff03678-8a48-4bdb-83c5-4941236f45bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 19:41:43.928371 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:43.928338    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 19:41:43.928662 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:43.928629    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9" podUID="7ca1424b-adf4-42f6-8165-44af258b508c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 19:41:44.659142 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:44.659087    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:44.836383 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:44.836342    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 19:41:44.836656 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:44.836627    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn" podUID="e56d7264-7d35-4dd8-946b-c1ab4d40e941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 19:41:44.936962 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:44.933857    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:41:44.936962 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:44.933930    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp" podUID="6ff03678-8a48-4bdb-83c5-4941236f45bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 19:41:44.946465 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:44.946424    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 19:41:44.946734 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:44.946703    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9" podUID="7ca1424b-adf4-42f6-8165-44af258b508c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 19:41:45.948669 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:45.948627    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:41:45.949340 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:45.949305    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp" podUID="6ff03678-8a48-4bdb-83c5-4941236f45bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 19:41:45.951325 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:45.951286    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 19:41:45.951518 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:45.951496    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9" podUID="7ca1424b-adf4-42f6-8165-44af258b508c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 19:41:45.976219 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:45.976178    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 19:41:46.002994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:46.002945    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn" podUID="e56d7264-7d35-4dd8-946b-c1ab4d40e941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 19:41:46.254972 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:46.254869    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:46.836293 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:46.836252    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 19:41:46.844018 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:46.843967    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn" podUID="e56d7264-7d35-4dd8-946b-c1ab4d40e941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 19:41:46.909860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:46.909645    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:41:46.909860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:46.909707    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp" podUID="6ff03678-8a48-4bdb-83c5-4941236f45bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 19:41:46.939861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:46.929288    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 19:41:46.939861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:46.929352    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9" podUID="7ca1424b-adf4-42f6-8165-44af258b508c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 19:41:47.835405 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:47.835333    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 19:41:47.836004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:47.835972    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn" podUID="e56d7264-7d35-4dd8-946b-c1ab4d40e941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 19:41:47.895293 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:47.895247    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:41:47.895650 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:47.895616    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp" podUID="6ff03678-8a48-4bdb-83c5-4941236f45bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 19:41:47.924099 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:47.924059    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 19:41:47.924322 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:47.924295    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9" podUID="7ca1424b-adf4-42f6-8165-44af258b508c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 19:41:48.053746 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:48.053716    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:48.890053 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:48.890013    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:41:48.890535 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:48.890084    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp" podUID="6ff03678-8a48-4bdb-83c5-4941236f45bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 19:41:48.924100 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:48.924061    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 19:41:48.924294 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:48.924118    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9" podUID="7ca1424b-adf4-42f6-8165-44af258b508c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 19:41:48.947443 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:48.947405    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:49.281262 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:49.281175    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:49.285820 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:49.285792    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:49.288936 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:49.288914    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:41:49.833378 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:49.833323    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:49.905658 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:49.905623    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:49.907153 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:49.907131    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:49.908088 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:49.908041    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:41:49.941485 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:49.941452    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:49.942068 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:49.942042    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:49.943059 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:49.943041    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:41:50.749197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:50.749123    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:51.438519 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:51.438487    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:52.122388 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:52.122348    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:53.105377 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:53.105342    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:53.832306 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:53.832267    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:54.613793 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:54.613739    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:54.981757 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:54.981665    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"]
Nov 03 19:41:54.987190 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:54.987162    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.000055 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.000006    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ef8f84\"/\"distinct-2-w0-ef8f84-kserve-self-signed-certs\""
Nov 03 19:41:55.005810 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.005781    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"]
Nov 03 19:41:55.022506 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.022468    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51fb02e-9d48-4c36-9911-1c077bcc915e-tls-certs\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.022916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.022885    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-model-cache\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.023063 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.022937    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-tmp-dir\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.023063 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.023035    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnd9r\" (UniqueName: \"kubernetes.io/projected/e51fb02e-9d48-4c36-9911-1c077bcc915e-kube-api-access-wnd9r\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.023183 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.023070    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-home\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.023183 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.023106    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-dshm\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.030673 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.030402    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"]
Nov 03 19:41:55.035267 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.035239    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.053526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.053499    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"]
Nov 03 19:41:55.058212 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.058172    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:55.081547 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.081512    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"]
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.134223    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-model-cache\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.134288    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-tmp-dir\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.134398    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg8lj\" (UniqueName: \"kubernetes.io/projected/e63a1c5a-826e-487e-b3eb-eab4278b8049-kube-api-access-wg8lj\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.134457    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnd9r\" (UniqueName: \"kubernetes.io/projected/e51fb02e-9d48-4c36-9911-1c077bcc915e-kube-api-access-wnd9r\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.134506    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.134542    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-home\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.134585    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.134614    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-dshm\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.134688    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.134721    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63a1c5a-826e-487e-b3eb-eab4278b8049-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.134793    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.134949    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.134990    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.135074    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51fb02e-9d48-4c36-9911-1c077bcc915e-tls-certs\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.135110    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.135694    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-model-cache\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.159458    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.201593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.160377    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-home\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.224281 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.223647    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-tmp-dir\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.283725 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.260367    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.284093 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.284060    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.284401 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.284382    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg8lj\" (UniqueName: \"kubernetes.io/projected/e63a1c5a-826e-487e-b3eb-eab4278b8049-kube-api-access-wg8lj\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.284638 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.284598    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.284811 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.284785    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.285046 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.285027    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.285181 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.285164    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63a1c5a-826e-487e-b3eb-eab4278b8049-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.287627 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.287605    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.288068 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.288046    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.292333 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.292284    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.293315 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.293282    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.300211 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.300186    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63a1c5a-826e-487e-b3eb-eab4278b8049-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.301374 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.301348    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.318639 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.318601    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.326380 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.326268    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.348903 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.337063    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-dshm\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.354010 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.349697    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg8lj\" (UniqueName: \"kubernetes.io/projected/e63a1c5a-826e-487e-b3eb-eab4278b8049-kube-api-access-wg8lj\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.376986 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.367535    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.376986 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.367758    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.376986 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.370335    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnd9r\" (UniqueName: \"kubernetes.io/projected/e51fb02e-9d48-4c36-9911-1c077bcc915e-kube-api-access-wnd9r\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.376986 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.374503    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51fb02e-9d48-4c36-9911-1c077bcc915e-tls-certs\") pod \"distinct-2-w0-ef8f84-kserve-569bf84859-rktsw\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") " pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.383475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.383450    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"]
Nov 03 19:41:55.393097 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.393049    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"]
Nov 03 19:41:55.404274 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.404226    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.404447 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.404396    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft6cn\" (UniqueName: \"kubernetes.io/projected/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-kube-api-access-ft6cn\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.404539 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.404484    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.404613 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.404563    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.404673 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.404609    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.404731 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.404679    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.404788 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.404728    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.404862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.404828    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.404949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.404928    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.415809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.415777    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.418733 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.418711    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"]
Nov 03 19:41:55.506394 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.506350    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.506586 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.506430    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.506586 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.506491    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.506586 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.506568    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft6cn\" (UniqueName: \"kubernetes.io/projected/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-kube-api-access-ft6cn\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.506755 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.506629    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.506755 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.506698    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.506755 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.506730    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.506962 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.506786    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.506962 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.506824    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.507071 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.507048    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.507218 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.507189    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.507288 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.507229    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.507361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.507337    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.507709 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.507671    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.509105 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.509081    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.509354 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.509330    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.519797 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.519766    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.519967 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.519949    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft6cn\" (UniqueName: \"kubernetes.io/projected/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-kube-api-access-ft6cn\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.606321 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.606278    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:55.608531 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.608498    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.608668 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.608536    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27821d19-33d6-4db9-a259-a7a555d03e15-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.608668 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.608560    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.608668 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.608657    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.608861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.608712    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.608861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.608775    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.608975 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.608900    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27821d19-33d6-4db9-a259-a7a555d03e15-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.608975 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.608924    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45chc\" (UniqueName: \"kubernetes.io/projected/27821d19-33d6-4db9-a259-a7a555d03e15-kube-api-access-45chc\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.608975 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.608963    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27821d19-33d6-4db9-a259-a7a555d03e15-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.632577 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.632135    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:55.669622 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.668286    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:55.692707 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.692400    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:55.717265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.710736    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.717265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.710793    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27821d19-33d6-4db9-a259-a7a555d03e15-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.717265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.710846    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.717265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.710883    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.717265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.710965    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.717265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.711013    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.717265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.711115    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27821d19-33d6-4db9-a259-a7a555d03e15-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.717265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.711148    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45chc\" (UniqueName: \"kubernetes.io/projected/27821d19-33d6-4db9-a259-a7a555d03e15-kube-api-access-45chc\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.717265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.711211    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27821d19-33d6-4db9-a259-a7a555d03e15-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.717265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.712039    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.717265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.712957    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.717265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.715524    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.717265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.717212    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27821d19-33d6-4db9-a259-a7a555d03e15-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.718282 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.718222    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.724320 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.723559    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.764060 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.754498    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27821d19-33d6-4db9-a259-a7a555d03e15-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.764060 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.754989    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45chc\" (UniqueName: \"kubernetes.io/projected/27821d19-33d6-4db9-a259-a7a555d03e15-kube-api-access-45chc\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.764060 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.763277    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27821d19-33d6-4db9-a259-a7a555d03e15-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:55.948378 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:55.948349    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"]
Nov 03 19:41:55.950036 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:41:55.950008    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode51fb02e_9d48_4c36_9911_1c077bcc915e.slice/crio-22d1b75f4d6716510a9496629f4f790aa054eae6101d27dc7c91c8ecb6f9485b WatchSource:0}: Error finding container 22d1b75f4d6716510a9496629f4f790aa054eae6101d27dc7c91c8ecb6f9485b: Status 404 returned error can't find the container with id 22d1b75f4d6716510a9496629f4f790aa054eae6101d27dc7c91c8ecb6f9485b
Nov 03 19:41:56.031292 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.031259    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"]
Nov 03 19:41:56.035041 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.035015    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:56.036877 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:41:56.036807    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63a1c5a_826e_487e_b3eb_eab4278b8049.slice/crio-fb0069ace7af2b7639c7a27e34996e2bbd9c74543c29bd91881949a5705d6b91 WatchSource:0}: Error finding container fb0069ace7af2b7639c7a27e34996e2bbd9c74543c29bd91881949a5705d6b91: Status 404 returned error can't find the container with id fb0069ace7af2b7639c7a27e34996e2bbd9c74543c29bd91881949a5705d6b91
Nov 03 19:41:56.042135 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.040407    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:56.042135 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.040507    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:56.042135 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.040560    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:56.055356 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.055096    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"]
Nov 03 19:41:56.055829 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:41:56.055801    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dacc3ed_4e8e_46d3_aa6c_7326bd160bfb.slice/crio-77bbe6e8626f1cd5c933ab8b61e2d237707bbd04f711fdcaf08680e48f54eaeb WatchSource:0}: Error finding container 77bbe6e8626f1cd5c933ab8b61e2d237707bbd04f711fdcaf08680e48f54eaeb: Status 404 returned error can't find the container with id 77bbe6e8626f1cd5c933ab8b61e2d237707bbd04f711fdcaf08680e48f54eaeb
Nov 03 19:41:56.062626 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.062597    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:56.062736 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.062661    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:56.062736 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.062694    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:56.356104 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.356060    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"]
Nov 03 19:41:56.356213 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:41:56.356043    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27821d19_33d6_4db9_a259_a7a555d03e15.slice/crio-452bd1692a8299225875010d0e153c00e8302a4afaf175c3418257ce7e5f8e52 WatchSource:0}: Error finding container 452bd1692a8299225875010d0e153c00e8302a4afaf175c3418257ce7e5f8e52: Status 404 returned error can't find the container with id 452bd1692a8299225875010d0e153c00e8302a4afaf175c3418257ce7e5f8e52
Nov 03 19:41:56.359063 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.359026    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:56.359150 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.359115    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:56.359201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.359166    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:41:56.515874 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.515819    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:56.624516 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.624425    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" event={"ID":"e63a1c5a-826e-487e-b3eb-eab4278b8049","Type":"ContainerStarted","Data":"36a30a9ec5c8248309e48633fbf8924b7ec2d1ff29a8eb2e1b39c94a10fe2bc2"}
Nov 03 19:41:56.624516 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.624491    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" event={"ID":"e63a1c5a-826e-487e-b3eb-eab4278b8049","Type":"ContainerStarted","Data":"fb0069ace7af2b7639c7a27e34996e2bbd9c74543c29bd91881949a5705d6b91"}
Nov 03 19:41:56.630412 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.630384    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" event={"ID":"e51fb02e-9d48-4c36-9911-1c077bcc915e","Type":"ContainerStarted","Data":"3142cabe30975b3bba06c15c4c82ef01c97fabe277331d3a9c264c2d5ad67e39"}
Nov 03 19:41:56.630600 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.630588    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" event={"ID":"e51fb02e-9d48-4c36-9911-1c077bcc915e","Type":"ContainerStarted","Data":"22d1b75f4d6716510a9496629f4f790aa054eae6101d27dc7c91c8ecb6f9485b"}
Nov 03 19:41:56.633134 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.633101    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" event={"ID":"27821d19-33d6-4db9-a259-a7a555d03e15","Type":"ContainerStarted","Data":"1bee9505297b475498c6691a54a9fe4fb618fe5b4438320854bee1b2728f6226"}
Nov 03 19:41:56.634004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.633978    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" event={"ID":"27821d19-33d6-4db9-a259-a7a555d03e15","Type":"ContainerStarted","Data":"452bd1692a8299225875010d0e153c00e8302a4afaf175c3418257ce7e5f8e52"}
Nov 03 19:41:56.636596 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.636091    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" event={"ID":"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb","Type":"ContainerStarted","Data":"78334ef3ed948b6a5413211648331cc6760a29bcb35e8e0645bb59022ff64c08"}
Nov 03 19:41:56.636596 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.636138    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" event={"ID":"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb","Type":"ContainerStarted","Data":"77bbe6e8626f1cd5c933ab8b61e2d237707bbd04f711fdcaf08680e48f54eaeb"}
Nov 03 19:41:56.657195 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.657100    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" podStartSLOduration=2.657083634 podStartE2EDuration="2.657083634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:56.656260065 +0000 UTC m=+872.910567291" watchObservedRunningTime="2026-11-03 19:41:56.657083634 +0000 UTC m=+872.911390927"
Nov 03 19:41:56.669935 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.669900    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:41:56.687450 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.687396    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" podStartSLOduration=1.687383042 podStartE2EDuration="1.687383042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:56.686253013 +0000 UTC m=+872.940560238" watchObservedRunningTime="2026-11-03 19:41:56.687383042 +0000 UTC m=+872.941690266"
Nov 03 19:41:56.692712 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.692687    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:41:56.707420 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.707363    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podStartSLOduration=2.707348489 podStartE2EDuration="2.707348489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:56.7064878 +0000 UTC m=+872.960795020" watchObservedRunningTime="2026-11-03 19:41:56.707348489 +0000 UTC m=+872.961655717"
Nov 03 19:41:56.741922 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:56.740997    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" podStartSLOduration=1.740977743 podStartE2EDuration="1.740977743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:56.730860627 +0000 UTC m=+872.985167850" watchObservedRunningTime="2026-11-03 19:41:56.740977743 +0000 UTC m=+872.995284968"
Nov 03 19:41:57.048659 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:57.048620    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:41:57.050020 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:57.049984    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:41:57.050151 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:57.050051    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:41:57.624861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:57.620985    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:41:57.624861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:57.621511    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:57.682307 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:57.682273    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:41:57.702868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:57.692084    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" podUID="e63a1c5a-826e-487e-b3eb-eab4278b8049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:41:57.719601 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:57.719562    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:41:57.719888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:57.719854    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:41:58.046858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:58.039759    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:41:58.046858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:58.039831    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:41:58.666521 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:58.666449    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:58.677868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:58.676266    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:41:58.677868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:58.676360    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" podUID="e63a1c5a-826e-487e-b3eb-eab4278b8049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:41:58.699869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:58.699739    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:41:58.699869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:58.699799    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:41:59.036250 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:59.036217    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:41:59.036510 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:59.036479    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:41:59.670313 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:59.670273    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:41:59.670597 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:59.670562    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" podUID="e63a1c5a-826e-487e-b3eb-eab4278b8049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:41:59.700302 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:59.700257    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:41:59.700945 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:41:59.700908    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:42:00.044785 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:00.044751    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:42:00.045057 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:00.045028    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:42:00.169403 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:00.168956    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:00.211780 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:00.211197    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:00.562980 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:00.562916    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:42:00.590436 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:00.590387    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:00.667064 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:00.616653    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:00.667941 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:00.660088    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:00.669966 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:00.669937    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:42:00.670129 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:00.670100    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" podUID="e63a1c5a-826e-487e-b3eb-eab4278b8049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:42:00.702041 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:00.701991    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:42:00.702687 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:00.702653    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:42:00.748959 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:00.748908    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded"
Nov 03 19:42:00.777051 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:00.777004    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:42:00.786398 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:00.786359    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:00.866701 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:00.866603    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:00.959988 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:00.959939    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:01.025878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.023945    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:01.046489 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.044338    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:42:01.046489 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.044402    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:42:01.216277 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.213604    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 19:42:01.216277 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.214409    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:42:01.257649 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.256929    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Nov 03 19:42:01.257649 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.257052    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:42:01.461866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.436589    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:01.461866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.445952    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 19:42:01.573360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.573318    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:42:01.574378 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.574353    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:42:01.691868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.686989    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:01.691868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.687436    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:01.691868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.688762    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:42:01.691868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.688825    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:42:01.691868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.689908    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 19:42:01.691868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.690263    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:42:01.691868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.691140    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:42:01.691868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.691176    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" podUID="e63a1c5a-826e-487e-b3eb-eab4278b8049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:42:01.766889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.725110    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:42:01.766889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.725170    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:42:01.785292 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.785247    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:01.788204 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.786789    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:01.816888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.788447    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:42:01.816888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.793920    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:42:01.963786 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.963681    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:01.975990 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:01.873816    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:02.040987 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.040944    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:02.064664 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.064171    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:42:02.064664 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.064259    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:42:02.064664 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.064293    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:42:02.064664 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.064328    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:42:02.132861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.121741    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:42:02.132861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.121803    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:42:02.228913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.216056    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:02.269866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.261988    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:02.290860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.270193    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:42:02.582957 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.582911    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:42:02.626907 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.626147    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:02.631203 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.631172    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:42:02.681532 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.681134    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:42:02.681532 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.681195    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" podUID="e63a1c5a-826e-487e-b3eb-eab4278b8049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:42:02.708615 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.703997    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 19:42:02.745902 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.711909    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 19:42:02.821869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.820945    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:42:02.821869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.821302    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:42:02.851693 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.832954    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:42:02.851693 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.833019    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:42:02.919258 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.919227    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:42:02.952348 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.952318    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:42:02.962032 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.953366    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:02.968710 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:02.967930    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:03.057465 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:03.057407    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:03.058444 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:03.058418    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:42:03.058695 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:03.058671    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:42:03.216401 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:03.216311    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 19:42:03.253110 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:03.253068    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:03.269615 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:03.269568    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:03.588272 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:03.587431    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:42:03.673001 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:03.672949    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:03.694862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:03.688259    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:42:03.694862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:03.688330    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" podUID="e63a1c5a-826e-487e-b3eb-eab4278b8049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:42:03.716862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:03.700338    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:42:03.716862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:03.700429    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:42:03.724013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:03.723962    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:03.731858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:03.724627    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 19:42:03.832961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:03.832201    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:03.955067 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:03.954945    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:04.024481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.024429    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:04.039860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.027069    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:42:04.062904 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.062827    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:04.075079 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.075027    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:42:04.075363 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.075329    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:42:04.223071 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.222965    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:04.371862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.327944    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:04.498462 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.498381    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:42:04.596024 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.595978    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:42:04.743987 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.743934    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:04.744554 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.744525    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:42:04.744756 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.744713    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:42:04.932866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.796481    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:04.932866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.796529    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:04.932866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.853699    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:42:04.932866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.874926    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:42:04.932866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.930394    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm" podUID="2b4a25ae-6965-4713-ab67-327317f678d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:04.932866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.923302    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:42:04.932866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.930466    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" podUID="e63a1c5a-826e-487e-b3eb-eab4278b8049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:42:04.966516 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.944467    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:42:04.966516 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.944546    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk" podUID="0398a799-4e11-4f8e-913f-8a42574c0ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:04.987876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:04.972673    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:05.051873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.037971    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:42:05.051873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.038497    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:42:05.051873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.040406    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:05.051873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.040686    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 19:42:05.085912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.079176    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:05.086933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.086812    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:42:05.086933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.086891    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:42:05.127654 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.127523    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:05.271020 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.233580    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:05.375087 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.375051    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:42:05.375369 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.375344    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj" podUID="9fea1109-4466-401f-8de1-3c9fae530f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:05.426141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.426096    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:05.649868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.631738    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:42:05.662585 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.662525    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:05.761867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.727847    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:42:05.761867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.727936    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:42:05.789978 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.769790    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 19:42:05.811000 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:05.810964    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:42:06.201867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.200371    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 19:42:06.202518 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.202482    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:06.214861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.204420    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:42:06.244460 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.216635    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:42:06.244460 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.217693    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:06.244460 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.217733    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:06.244460 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.231234    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:06.244460 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.236529    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:06.244460 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.236820    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:42:06.265574 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.263039    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:42:06.265574 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.263104    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:42:06.265574 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.264526    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:42:06.265574 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.264568    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" podUID="e63a1c5a-826e-487e-b3eb-eab4278b8049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:42:06.334870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.334820    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:42:06.335127 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.335110    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:42:06.650861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.647574    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:42:06.704860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.694955    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:06.704860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.698062    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Nov 03 19:42:06.704860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.698168    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:42:06.704860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.698195    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:42:06.704860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.698314    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:42:06.704860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.698339    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" podUID="e63a1c5a-826e-487e-b3eb-eab4278b8049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:42:06.772863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.771791    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:06.853870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.852664    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:06.976059 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:06.975969    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:07.135576 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.135541    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:42:07.151764 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.151725    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:42:07.202352 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.202293    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:07.248015 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.245562    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:07.248015 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.246066    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:42:07.248015 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.246476    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:42:07.248015 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.246740    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 19:42:07.248015 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.247009    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 19:42:07.250327 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.248808    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:42:07.251569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.251539    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:07.376866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.376515    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:42:07.414632 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.414335    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:07.452868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.452480    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:42:07.543922 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.519289    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:42:07.665013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.664984    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:42:07.721867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.718265    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:07.757891 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.757857    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:42:07.785871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.784938    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:42:07.801952 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.796531    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 19:42:07.801952 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.796941    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:42:07.836878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.834005    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:42:07.836878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.834062    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" podUID="e63a1c5a-826e-487e-b3eb-eab4278b8049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:42:07.865487 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.864764    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:42:07.924868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:07.923011    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:42:08.035811 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:08.035767    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:42:08.036070 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:08.036036    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:42:08.060234 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:08.050110    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:42:08.060234 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:08.060229    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:42:08.060518 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:08.060279    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:42:08.061387 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:08.060801    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:42:08.075088 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:08.067097    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:42:08.148777 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:08.148747    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:42:08.418699 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:08.418600    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:08.670081 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:08.669984    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:42:08.670081 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:08.670050    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" podUID="e63a1c5a-826e-487e-b3eb-eab4278b8049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:42:08.710623 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:08.696184    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:42:08.710623 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:08.696248    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:42:09.036216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:09.036173    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:42:09.036391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:09.036240    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:42:09.448101 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:09.446435    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:09.868422 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:09.867537    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:42:09.868422 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:09.868257    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:42:09.869924 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:09.869870    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:42:10.049420 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:10.049377    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:42:10.050291 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:10.050268    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:42:10.051006 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:10.050987    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:42:10.237391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:10.237302    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:10.693869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:10.693693    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:42:10.735036 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:10.734999    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:42:10.735321 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:10.735289    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:11.249867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:11.247608    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:11.416873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:11.416788    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:42:11.486901 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:11.486824    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:42:11.597782 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:11.597741    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:11.744059 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:11.744019    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:11.940513 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:11.940427    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:12.604175 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:12.604133    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:13.261176 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:42:13.261141    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:42:13.319941 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:13.319904    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:14.036216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:14.036182    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:14.780024 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:14.779971    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:15.742389 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:15.742351    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:16.836859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:16.825144    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:17.658096 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:17.658042    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:18.423859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:18.420858    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:19.139726 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:19.139692    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:19.844777 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:19.843064    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:20.623115 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:20.623062    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:21.451951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:21.451913    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:22.124954 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:22.124918    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:22.724419 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:22.724365    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:23.414875 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:23.414814    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:24.207741 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.207708    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:24.687735 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.687698    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"]
Nov 03 19:42:24.695889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.695806    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.707998 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.707964    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"]
Nov 03 19:42:24.754765 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.754730    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.754969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.754806    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd104015-ca9a-4193-bf07-f26376a66366-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.754969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.754919    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.755085 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.754981    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd104015-ca9a-4193-bf07-f26376a66366-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.755085 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.755065    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dchcz\" (UniqueName: \"kubernetes.io/projected/bd104015-ca9a-4193-bf07-f26376a66366-kube-api-access-dchcz\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.755195 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.755132    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.755195 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.755162    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.755282 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.755226    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd104015-ca9a-4193-bf07-f26376a66366-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.755347 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.755329    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.856692 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.856659    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.856898 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.856798    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.856898 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.856881    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd104015-ca9a-4193-bf07-f26376a66366-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.857030 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.856930    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.857030 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.857004    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd104015-ca9a-4193-bf07-f26376a66366-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.857140 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.857036    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dchcz\" (UniqueName: \"kubernetes.io/projected/bd104015-ca9a-4193-bf07-f26376a66366-kube-api-access-dchcz\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.857140 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.857098    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.857140 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.857126    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.857274 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.857133    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.857335 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.857315    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.857464 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.857410    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd104015-ca9a-4193-bf07-f26376a66366-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.857611 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.857586    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.857673 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.857622    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.858217 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.858190    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd104015-ca9a-4193-bf07-f26376a66366-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.859096 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.859071    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.859704 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.859684    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd104015-ca9a-4193-bf07-f26376a66366-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.870071 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.870048    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dchcz\" (UniqueName: \"kubernetes.io/projected/bd104015-ca9a-4193-bf07-f26376a66366-kube-api-access-dchcz\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.870226 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.870153    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd104015-ca9a-4193-bf07-f26376a66366-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:24.912465 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:24.912430    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:25.027904 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.027205    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:25.320589 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.320558    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"]
Nov 03 19:42:25.321748 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:42:25.321721    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd104015_ca9a_4193_bf07_f26376a66366.slice/crio-141c27d75637a581748c364f49215e632a34a2a3f6b557c4ddf2219ebff9b0d1 WatchSource:0}: Error finding container 141c27d75637a581748c364f49215e632a34a2a3f6b557c4ddf2219ebff9b0d1: Status 404 returned error can't find the container with id 141c27d75637a581748c364f49215e632a34a2a3f6b557c4ddf2219ebff9b0d1
Nov 03 19:42:25.324471 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.324441    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:42:25.324554 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.324507    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:42:25.324554 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.324538    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:42:25.413183 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.413153    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"]
Nov 03 19:42:25.419007 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.418972    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.444380 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.443636    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"]
Nov 03 19:42:25.481128 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.481093    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.481312 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.481148    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e9f189-e173-4b40-b4a2-3455be004378-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.481312 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.481285    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.481401 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.481319    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.481443 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.481391    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cmfd\" (UniqueName: \"kubernetes.io/projected/22e9f189-e173-4b40-b4a2-3455be004378-kube-api-access-9cmfd\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.481479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.481454    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.481531 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.481508    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.481574 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.481559    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e9f189-e173-4b40-b4a2-3455be004378-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.481617 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.481593    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e9f189-e173-4b40-b4a2-3455be004378-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.583090 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.582984    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.583090 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.583064    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.583317 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.583118    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e9f189-e173-4b40-b4a2-3455be004378-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.583317 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.583150    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e9f189-e173-4b40-b4a2-3455be004378-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.583421 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.583371    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.583421 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.583412    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e9f189-e173-4b40-b4a2-3455be004378-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.583523 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.583449    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.583523 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.583479    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.583523 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.583510    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cmfd\" (UniqueName: \"kubernetes.io/projected/22e9f189-e173-4b40-b4a2-3455be004378-kube-api-access-9cmfd\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.583932 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.583908    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.584229 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.584177    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.584319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.584224    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.584468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.584446    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.585265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.585239    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e9f189-e173-4b40-b4a2-3455be004378-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.586246 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.586223    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.586756 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.586732    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" event={"ID":"bd104015-ca9a-4193-bf07-f26376a66366","Type":"ContainerStarted","Data":"4222219fb4ec3ee20b71b4e5f53a26d0b6ee4e14f050a2b9be9302feba69f1b4"}
Nov 03 19:42:25.586884 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.586787    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" event={"ID":"bd104015-ca9a-4193-bf07-f26376a66366","Type":"ContainerStarted","Data":"141c27d75637a581748c364f49215e632a34a2a3f6b557c4ddf2219ebff9b0d1"}
Nov 03 19:42:25.587078 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.587059    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e9f189-e173-4b40-b4a2-3455be004378-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.596108 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.596082    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cmfd\" (UniqueName: \"kubernetes.io/projected/22e9f189-e173-4b40-b4a2-3455be004378-kube-api-access-9cmfd\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.596194 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.596121    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e9f189-e173-4b40-b4a2-3455be004378-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:25.610364 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.610302    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" podStartSLOduration=1.6102853339999998 podStartE2EDuration="1.610285334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:25.609814543 +0000 UTC m=+901.864121773" watchObservedRunningTime="2026-11-03 19:42:25.610285334 +0000 UTC m=+901.864592560"
Nov 03 19:42:25.724612 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.724578    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:25.769109 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:25.769071    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:26.027470 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:26.027436    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:26.028509 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:26.028476    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:42:26.028627 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:26.028539    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" podUID="bd104015-ca9a-4193-bf07-f26376a66366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 19:42:26.158635 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:26.157299    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"]
Nov 03 19:42:26.160541 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:42:26.160512    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e9f189_e173_4b40_b4a2_3455be004378.slice/crio-b39852c4d07a5cf2e64eaeb44be5f25ff47eaac24393948b3dff2a29800addf9 WatchSource:0}: Error finding container b39852c4d07a5cf2e64eaeb44be5f25ff47eaac24393948b3dff2a29800addf9: Status 404 returned error can't find the container with id b39852c4d07a5cf2e64eaeb44be5f25ff47eaac24393948b3dff2a29800addf9
Nov 03 19:42:26.194000 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:26.168313    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:42:26.194000 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:26.168458    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:42:26.194000 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:26.168539    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:42:26.643260 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:26.643222    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:26.662932 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:26.662329    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" event={"ID":"22e9f189-e173-4b40-b4a2-3455be004378","Type":"ContainerStarted","Data":"c932bcc9628f0f0e01ee8e29f48e5d96f4d00e307ac8d0cca6023982b936e188"}
Nov 03 19:42:26.662932 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:26.662538    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" event={"ID":"22e9f189-e173-4b40-b4a2-3455be004378","Type":"ContainerStarted","Data":"b39852c4d07a5cf2e64eaeb44be5f25ff47eaac24393948b3dff2a29800addf9"}
Nov 03 19:42:26.698858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:26.698645    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" podStartSLOduration=1.698626438 podStartE2EDuration="1.698626438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:26.696652779 +0000 UTC m=+902.950960009" watchObservedRunningTime="2026-11-03 19:42:26.698626438 +0000 UTC m=+902.952933664"
Nov 03 19:42:26.772898 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:26.769688    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:27.028378 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:27.028270    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:42:27.028378 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:27.028330    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" podUID="bd104015-ca9a-4193-bf07-f26376a66366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 19:42:27.716870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:27.716696    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:27.795859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:27.789976    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:42:27.795859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:27.790041    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" podUID="22e9f189-e173-4b40-b4a2-3455be004378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:42:28.029312 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:28.029224    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:42:28.029312 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:28.029287    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" podUID="bd104015-ca9a-4193-bf07-f26376a66366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 19:42:28.667290 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:28.667178    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:28.774865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:28.771236    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:42:28.774865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:28.771300    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" podUID="22e9f189-e173-4b40-b4a2-3455be004378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:42:29.028470 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:29.028363    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:42:29.028470 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:29.028422    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" podUID="bd104015-ca9a-4193-bf07-f26376a66366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 19:42:29.772317 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:29.772275    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:42:29.772613 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:29.772580    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" podUID="22e9f189-e173-4b40-b4a2-3455be004378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:42:30.037508 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.037472    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:42:30.053862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.044912    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" podUID="bd104015-ca9a-4193-bf07-f26376a66366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 19:42:30.089463 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.089427    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:30.127064 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.127007    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:30.200468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.200428    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:30.201174 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.201141    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:30.222489 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.222449    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:30.252024 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.251966    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:30.253679 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.253651    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:30.370142 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.370010    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:30.443860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.438787    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:30.606346 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.606278    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:30.692569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.692468    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:30.718684 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.718645    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded"
Nov 03 19:42:30.770363 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.770326    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:42:30.770655 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.770627    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" podUID="22e9f189-e173-4b40-b4a2-3455be004378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:42:30.897910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.893930    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:30.976940 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:30.973609    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:42:31.030857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.030172    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:42:31.030857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.030237    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" podUID="bd104015-ca9a-4193-bf07-f26376a66366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 19:42:31.175877 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.174928    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:31.204870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.201935    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:31.204870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.202771    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:42:31.253142 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.253036    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:31.253807 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.253755    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:42:31.254189 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.254152    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 19:42:31.256406 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.256370    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 19:42:31.288094 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.256963    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:42:31.418404 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.418376    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:42:31.429871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.386380    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:31.429871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.428978    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:42:31.469880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.458939    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:31.469880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.464477    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:42:31.617888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.606537    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:31.812532 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.812493    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:42:31.812825 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.812794    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" podUID="22e9f189-e173-4b40-b4a2-3455be004378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:42:31.846748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.846697    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:42:31.900163 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.900060    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:31.990882 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:31.984498    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:32.046861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.035148    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:42:32.046861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.035226    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" podUID="bd104015-ca9a-4193-bf07-f26376a66366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 19:42:32.083085 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.083040    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:32.193211 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.193106    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:42:32.196262 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.196234    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:42:32.196724 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.196705    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:42:32.491267 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.491177    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:42:32.505687 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.261080    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:32.507175 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.287290    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:32.507486 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.287347    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:32.508713 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.429459    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:32.509789 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.212709    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:32.606206 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.606163    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:32.645026 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.644976    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:32.699859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.690291    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:32.776068 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.775967    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:42:32.776349 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.776305    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" podUID="22e9f189-e173-4b40-b4a2-3455be004378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:42:32.848007 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.847943    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:32.852865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.851946    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:42:32.902020 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.901978    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:32.997912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:32.997862    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:33.029103 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.028306    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:42:33.029718 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.029684    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" podUID="bd104015-ca9a-4193-bf07-f26376a66366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 19:42:33.105869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.101951    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:33.121867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.106812    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:42:33.215874 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.201700    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:33.267863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.262926    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:33.511999 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.511946    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:33.512770 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.512736    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:42:33.513268 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.513242    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:42:33.591955 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.589028    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:33.591955 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.590825    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 19:42:33.634865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.634375    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:33.675858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.673133    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:42:33.715858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.713971    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:42:33.716344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.716310    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 19:42:33.718233 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.718212    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:42:33.844231 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.844192    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:42:33.844514 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:33.844472    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" podUID="22e9f189-e173-4b40-b4a2-3455be004378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:42:34.009787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.009727    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:34.010771 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.010718    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:42:34.201876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.201747    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:34.303866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.303503    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:42:34.303866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.303588    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" podUID="bd104015-ca9a-4193-bf07-f26376a66366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 19:42:34.416857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.409172    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:34.464480 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.447149    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:42:34.572454 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.572406    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:42:34.595190 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.595154    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:34.595754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.595723    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:42:34.596211 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.596186    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:42:34.596702 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.596678    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 19:42:34.636096 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.636033    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:42:34.771931 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.724825    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:42:34.771931 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.729671    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:42:34.827204 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.827111    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:42:34.827204 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.827173    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" podUID="22e9f189-e173-4b40-b4a2-3455be004378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:42:34.857008 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.847683    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:42:34.857008 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.855254    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:42:35.002708 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.962318    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:42:35.002708 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:34.974587    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:42:35.005369 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:35.005344    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:42:35.010456 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:35.010434    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:42:35.010746 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:35.010731    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:42:35.014855 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:35.010880    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:42:35.014855 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:35.013521    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:42:35.014855 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:35.013559    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:42:35.014855 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:35.013619    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:42:35.031995 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:35.031404    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:42:35.031995 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:35.031457    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" podUID="bd104015-ca9a-4193-bf07-f26376a66366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 19:42:35.519352 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:35.519315    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:35.774144 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:35.773422    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:42:35.774144 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:35.773489    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" podUID="22e9f189-e173-4b40-b4a2-3455be004378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:42:36.270982 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:36.270946    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:36.272702 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:36.271820    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:36.276726 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:36.276700    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:42:36.537866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:36.535660    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:36.770770 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:36.770739    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:42:36.770965 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:36.770795    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" podUID="22e9f189-e173-4b40-b4a2-3455be004378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:42:37.437873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:37.425133    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:37.775055 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:37.774977    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:37.775451 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:37.775412    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:37.776144 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:37.776122    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:42:38.230031 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:38.229987    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:39.031806 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:39.031762    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:39.764859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:39.763651    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:40.537652 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:40.537614    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:41.668864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:41.664868    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:42.322586 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:42.322552    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:43.036519 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:43.036468    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:43.256121 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:42:43.256087    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:42:43.860476 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:43.860440    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:44.484342 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:44.484306    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:45.173068 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:45.162371    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:45.954859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:45.945937    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:46.718197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:46.718158    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:47.394902 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:47.394700    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:48.227188 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:48.227150    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:49.220515 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:49.220464    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:49.943952 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:49.938414    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:50.909244 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:50.909203    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:51.603656 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:51.603617    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:52.342334 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:52.342297    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:52.989241 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:52.989205    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:53.925237 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:53.925201    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:54.845866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:54.842746    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:54.937287 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:54.937253    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"]
Nov 03 19:42:54.941443 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:54.941422    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:54.953526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:54.953498    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"]
Nov 03 19:42:55.068911 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.068878    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.069089 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.069016    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbwkl\" (UniqueName: \"kubernetes.io/projected/a48f6c83-ae97-4927-9d94-2d17fd679536-kube-api-access-jbwkl\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.069089 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.069060    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.069214 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.069090    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.069214 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.069160    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.069364 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.069344    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.069443 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.069411    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.069496 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.069463    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a48f6c83-ae97-4927-9d94-2d17fd679536-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.069540 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.069509    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.170806 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.170716    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.170806 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.170755    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a48f6c83-ae97-4927-9d94-2d17fd679536-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.170806 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.170786    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.171123 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.170891    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.171123 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.170979    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbwkl\" (UniqueName: \"kubernetes.io/projected/a48f6c83-ae97-4927-9d94-2d17fd679536-kube-api-access-jbwkl\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.171123 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.171030    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.171123 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.171070    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.171123 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.171115    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.171367 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.171296    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.171425 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.171404    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.171804 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.171677    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a48f6c83-ae97-4927-9d94-2d17fd679536-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.171804 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.171752    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.172012 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.171928    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.172187 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.172167    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.174084 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.174053    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.174307 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.174282    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.185497 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.185474    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.185497 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.185484    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbwkl\" (UniqueName: \"kubernetes.io/projected/a48f6c83-ae97-4927-9d94-2d17fd679536-kube-api-access-jbwkl\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.257138 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.257107    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:55.297493 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.297460    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"]
Nov 03 19:42:55.302646 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.302619    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.332020 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.327186    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"]
Nov 03 19:42:55.373933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.373716    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.373933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.373815    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.373933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.373867    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.373933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.373900    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.373933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.373945    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1faf58-6636-4c3c-a702-53c2fe91d649-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.374344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.373992    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlrg7\" (UniqueName: \"kubernetes.io/projected/9c1faf58-6636-4c3c-a702-53c2fe91d649-kube-api-access-wlrg7\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.374344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.374032    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.374344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.374099    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.374344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.374122    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.475051 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.474952    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.475051 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.475004    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.475051 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.475033    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.475332 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.475076    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.475332 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.475126    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1faf58-6636-4c3c-a702-53c2fe91d649-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.475332 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.475190    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlrg7\" (UniqueName: \"kubernetes.io/projected/9c1faf58-6636-4c3c-a702-53c2fe91d649-kube-api-access-wlrg7\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.475332 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.475227    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.475332 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.475325    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.475721 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.475361    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.475877 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.475821    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.476261 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.476235    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1faf58-6636-4c3c-a702-53c2fe91d649-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.476409 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.476364    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.476409 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.476387    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.476655 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.476628    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.478129 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.478109    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.478287 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.478270    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.488524 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.488498    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlrg7\" (UniqueName: \"kubernetes.io/projected/9c1faf58-6636-4c3c-a702-53c2fe91d649-kube-api-access-wlrg7\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.488637 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.488557    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.560464 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.560436    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"]
Nov 03 19:42:55.561503 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:42:55.561480    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48f6c83_ae97_4927_9d94_2d17fd679536.slice/crio-d4f7adaa7118592118a73af2f73d4d17d71f4bbfe3f4f019fc0790f111adff92 WatchSource:0}: Error finding container d4f7adaa7118592118a73af2f73d4d17d71f4bbfe3f4f019fc0790f111adff92: Status 404 returned error can't find the container with id d4f7adaa7118592118a73af2f73d4d17d71f4bbfe3f4f019fc0790f111adff92
Nov 03 19:42:55.564203 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.564170    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:42:55.564295 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.564234    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:42:55.564295 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.564268    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:42:55.656821 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.655052    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:55.660859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.659815    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:55.771828 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.770539    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" event={"ID":"a48f6c83-ae97-4927-9d94-2d17fd679536","Type":"ContainerStarted","Data":"ebdeea957a9ae3743cc865dc0892088979ccb6a2f363a772e786ef8f52816fe1"}
Nov 03 19:42:55.771828 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.770618    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" event={"ID":"a48f6c83-ae97-4927-9d94-2d17fd679536","Type":"ContainerStarted","Data":"d4f7adaa7118592118a73af2f73d4d17d71f4bbfe3f4f019fc0790f111adff92"}
Nov 03 19:42:55.843158 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:55.842156    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podStartSLOduration=1.8421294700000002 podStartE2EDuration="1.84212947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:55.837960746 +0000 UTC m=+932.092267974" watchObservedRunningTime="2026-11-03 19:42:55.84212947 +0000 UTC m=+932.096436695"
Nov 03 19:42:56.091381 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:56.091351    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"]
Nov 03 19:42:56.092553 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:42:56.092528    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1faf58_6636_4c3c_a702_53c2fe91d649.slice/crio-3d94939e6b623df883af7ecb5f7089719da62b20e713c035214bb3407069b187 WatchSource:0}: Error finding container 3d94939e6b623df883af7ecb5f7089719da62b20e713c035214bb3407069b187: Status 404 returned error can't find the container with id 3d94939e6b623df883af7ecb5f7089719da62b20e713c035214bb3407069b187
Nov 03 19:42:56.095477 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:56.095444    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:42:56.095579 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:56.095530    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:42:56.095649 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:56.095581    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:42:56.258147 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:56.258113    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:42:56.259383 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:56.259354    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:42:56.259651 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:56.259411    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:42:56.666911 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:56.664002    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:56.810375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:56.810335    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" event={"ID":"9c1faf58-6636-4c3c-a702-53c2fe91d649","Type":"ContainerStarted","Data":"f72f1dad6fc491872ef0294081f6f0c7bd7b8a35abcdf98cdb4bc3da8fad8ede"}
Nov 03 19:42:56.810577 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:56.810432    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" event={"ID":"9c1faf58-6636-4c3c-a702-53c2fe91d649","Type":"ContainerStarted","Data":"3d94939e6b623df883af7ecb5f7089719da62b20e713c035214bb3407069b187"}
Nov 03 19:42:56.851008 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:56.850951    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" podStartSLOduration=1.850932921 podStartE2EDuration="1.850932921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:56.846205403 +0000 UTC m=+933.100512641" watchObservedRunningTime="2026-11-03 19:42:56.850932921 +0000 UTC m=+933.105240145"
Nov 03 19:42:57.258186 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:57.258157    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:42:57.258574 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:57.258217    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:42:57.456098 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:57.456037    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:57.672859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:57.663110    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:42:57.682856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:57.676066    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:42:57.682856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:57.676142    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:42:58.257562 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:58.257525    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:42:58.257743 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:58.257594    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:42:58.550861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:58.541651    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:58.663100 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:58.663063    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:42:58.663385 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:58.663355    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:42:59.257559 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:59.257510    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:42:59.257777 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:59.257584    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:42:59.553479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:59.552982    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:59.716858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:59.714012    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:42:59.716858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:42:59.714161    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:43:00.259292 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:00.259256    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:43:00.259594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:00.259563    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:43:00.665867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:00.661625    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:43:00.665867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:00.661692    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:43:01.333871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:01.273918    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Nov 03 19:43:01.377969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:01.377922    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:43:01.378253 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:01.378212    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:43:01.519864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:01.517475    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:01.519864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:01.517785    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:43:01.608915 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:01.595954    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:01.608915 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:01.602949    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 19:43:01.658855 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:01.658222    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:43:01.658855 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:01.658286    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:43:01.908044 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:01.907944    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:02.018990 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.018938    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:02.019699 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.019322    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:02.040202 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.040156    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:02.193497 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.193403    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:02.248087 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.248030    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:02.262015 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.261976    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:43:02.262308 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.262278    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:43:02.411795 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.411735    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:02.485927 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.474915    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:43:02.532053 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.532002    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:43:02.532520 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.532325    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:02.573050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.540564    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:43:02.601039 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.559140    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:02.631970 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.606009    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:02.632701 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.632677    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:43:02.672890 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.600467    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:02.674023 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.673983    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:43:02.743949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.706045    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:43:02.743949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.672063    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:02.743949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.725305    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:43:02.743949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.725360    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:43:02.950694 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.950650    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:02.951627 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:02.951600    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:43:03.031550 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.031454    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:03.094879 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.031762    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:43:03.094879 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.085624    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:43:03.094879 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.085691    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:43:03.094879 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.046969    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:03.094879 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.088441    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:43:03.257008 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.256957    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded"
Nov 03 19:43:03.328223 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.286995    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:03.331921 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.325370    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:43:03.356860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.353112    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:43:03.356860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.332186    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:43:03.420002 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.418759    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:03.422023 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.420795    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:43:03.505371 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.505327    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:03.507326 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.507301    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:43:03.542984 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.542946    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:03.635334 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.635229    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:03.637322 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.637294    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:43:03.639241 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.639211    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:03.714918 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.690036    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:43:03.715248 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.715217    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:43:03.732207 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.693158    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:43:03.732605 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.690106    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:03.732958 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.732807    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:03.735186 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:03.733332    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:43:04.010226 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.010125    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:04.099474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.099428    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:04.312676 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.312627    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:43:04.351897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.337104    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:43:04.351897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.337026    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:43:04.351897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.336723    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:43:04.419238 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.419191    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 19:43:04.438171 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.438119    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 19:43:04.562429 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.562386    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:43:04.682888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.655442    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:04.682888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.669219    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:43:04.755073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.755011    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:04.778777 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.764465    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:04.789451 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.789407    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:43:04.791258 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.791230    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:43:04.791461 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.791436    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:43:04.847279 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.845050    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:43:04.847279 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.845134    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk" podUID="0398a799-4e11-4f8e-913f-8a42574c0ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:04.975922 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:04.965443    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:05.014536 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.014488    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:05.109646 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.109604    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:05.119994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.119947    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:43:05.261557 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.261460    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:43:05.261826 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.261791    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:43:05.320511 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.320476    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:43:05.456859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.424109    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:05.457271 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.431929    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:05.499867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.431990    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:43:05.570857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.512062    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:43:05.570857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.570313    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm" podUID="e6a4dcf8-18ac-443b-b78b-014a4e38ce34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:05.570857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.551938    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:43:05.570857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.570386    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn" podUID="e56d7264-7d35-4dd8-946b-c1ab4d40e941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:05.670862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.664877    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:43:05.670862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.664937    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:43:05.755001 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.754948    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:05.756363 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.756326    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:05.757261 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.757234    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:05.765394 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.765357    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:43:05.766022 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.765600    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:43:05.799119 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.799072    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:05.998665 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.998555    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:06.019133 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:05.999576    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:43:06.048861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.031111    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp" podUID="6ff03678-8a48-4bdb-83c5-4941236f45bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:06.050763 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.016647    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:06.099862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.087092    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:43:06.099862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.087162    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9" podUID="7ca1424b-adf4-42f6-8165-44af258b508c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:06.133988 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.121913    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:06.260321 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.260227    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:43:06.260581 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.260549    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:43:06.358959 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.358912    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:06.459931 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.457946    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:43:06.459931 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.458379    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:06.585867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.583106    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:06.587467 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.587345    5237 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:43:06.587467 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.587404    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc" podUID="e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:06.638455 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.638360    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Nov 03 19:43:06.650257 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.650216    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Nov 03 19:43:06.664739 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.664694    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:43:06.665452 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.665007    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:43:06.738450 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.738381    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded"
Nov 03 19:43:06.759986 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.759937    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:06.760580 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.760544    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:43:06.761005 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.760982    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 19:43:06.765785 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.765758    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:43:06.774730 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.774421    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:06.814416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.814039    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:43:06.890250 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:06.890140    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:07.013978 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.013929    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:07.051749 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.051085    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:07.144053 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.143941    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:07.265870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.265248    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:43:07.265870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.265317    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:43:07.360447 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.360364    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:07.361314 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.361291    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:43:07.433927 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.433756    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:07.487860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.461447    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:07.487860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.461820    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:07.584890 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.584815    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:07.664280 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.664241    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:43:07.664540 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.664511    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:43:07.695005 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.671974    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:43:07.695271 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.695235    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" podUID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:07.767891 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.767068    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:43:07.767891 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.767674    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:07.768164 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.767988    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 19:43:07.775862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.768233    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:43:07.791871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.791517    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:07.831183 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:07.820403    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:08.015058 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.014950    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 19:43:08.053047 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.053001    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:08.053991 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.053962    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded"
Nov 03 19:43:08.153405 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.153355    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:08.257491 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.257453    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:43:08.257742 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.257710    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:43:08.364861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.364430    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:08.480197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.480146    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:08.480825 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.480792    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:08.501737 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.501694    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:08.587492 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.587303    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:43:08.683926 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.670713    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:43:08.683926 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.670774    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:43:08.787870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.778065    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:43:08.787870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.778901    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:43:08.787870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.779181    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 19:43:08.787870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.779491    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:08.793387 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.793340    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:08.821901 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.821282    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:08.851861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.848146    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:43:08.851861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:08.848376    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" podUID="22e9f189-e173-4b40-b4a2-3455be004378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:09.020023 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.017937    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 19:43:09.055860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.053703    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:09.160856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.159984    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:09.260350 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.260309    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:43:09.260646 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.260615    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:43:09.317864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.308148    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:09.370137 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.370078    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 19:43:09.488530 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.488480    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 19:43:09.506466 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.506419    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:09.592121 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.592018    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:43:09.607729 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.604107    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:43:09.801169 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.798447    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:09.833488 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.831090    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:43:09.833488 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.832536    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:43:09.833488 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.832583    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:43:09.861932 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.834269    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:43:09.861932 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.834188    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 19:43:09.870030 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.869991    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:09.886555 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.882111    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:09.929687 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.929576    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:43:09.930101 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.929994    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:43:09.930101 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.930075    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:43:09.930420 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.930373    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:43:09.954722 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:09.954677    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:43:10.015975 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:10.013418    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:43:10.019471 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:10.019444    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:43:10.021188 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:10.021166    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:43:10.054187 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:10.049661    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:43:10.085642 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:10.085473    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:43:10.091197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:10.091172    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:43:10.257958 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:10.257876    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:43:10.258211 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:10.258180    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:43:10.258678 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:10.258648    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:43:10.617860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:10.617350    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:10.665156 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:10.661957    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:43:10.665156 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:10.662738    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:43:10.665156 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:10.662786    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:43:11.534337 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:11.534296    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:12.167709 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:12.167667    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7107-729a69"]
Nov 03 19:43:12.195098 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:12.195066    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7107-729a69"
Nov 03 19:43:12.204773 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:12.204659    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7107-729a69"]
Nov 03 19:43:12.224376 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:12.224349    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hjqcx\""
Nov 03 19:43:12.296549 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:12.296507    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xppnv\" (UniqueName: \"kubernetes.io/projected/1c8bc351-3ae6-4189-bba2-f9d1b4230530-kube-api-access-xppnv\") pod \"test-curl-7107-729a69\" (UID: \"1c8bc351-3ae6-4189-bba2-f9d1b4230530\") " pod="opendatahub/test-curl-7107-729a69"
Nov 03 19:43:12.309994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:12.309957    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:43:12.310675 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:12.310644    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:12.397900 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:12.397829    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xppnv\" (UniqueName: \"kubernetes.io/projected/1c8bc351-3ae6-4189-bba2-f9d1b4230530-kube-api-access-xppnv\") pod \"test-curl-7107-729a69\" (UID: \"1c8bc351-3ae6-4189-bba2-f9d1b4230530\") " pod="opendatahub/test-curl-7107-729a69"
Nov 03 19:43:12.417431 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:12.417398    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xppnv\" (UniqueName: \"kubernetes.io/projected/1c8bc351-3ae6-4189-bba2-f9d1b4230530-kube-api-access-xppnv\") pod \"test-curl-7107-729a69\" (UID: \"1c8bc351-3ae6-4189-bba2-f9d1b4230530\") " pod="opendatahub/test-curl-7107-729a69"
Nov 03 19:43:12.492745 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:12.492652    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:43:12.497021 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:12.496493    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:43:12.544858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:12.543799    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:12.566702 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:12.563494    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7107-729a69"
Nov 03 19:43:12.719874 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:12.716109    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:43:12.719874 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:12.716178    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:13.046344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:13.046307    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:43:13.051413 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:13.051380    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:43:13.074397 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:13.074370    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7107-729a69"]
Nov 03 19:43:13.076622 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:43:13.076589    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c8bc351_3ae6_4189_bba2_f9d1b4230530.slice/crio-14cd4a27ee6861d533e7a2c904aeaf0db5bfe149e27c1da447a5d3d00de9818c WatchSource:0}: Error finding container 14cd4a27ee6861d533e7a2c904aeaf0db5bfe149e27c1da447a5d3d00de9818c: Status 404 returned error can't find the container with id 14cd4a27ee6861d533e7a2c904aeaf0db5bfe149e27c1da447a5d3d00de9818c
Nov 03 19:43:13.078681 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:13.078662    5237 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:43:13.127129 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:13.127047    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7107-729a69" event={"ID":"1c8bc351-3ae6-4189-bba2-f9d1b4230530","Type":"ContainerStarted","Data":"14cd4a27ee6861d533e7a2c904aeaf0db5bfe149e27c1da447a5d3d00de9818c"}
Nov 03 19:43:13.260164 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:43:13.260130    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:43:13.351711 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:13.351636    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:14.191869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:14.180579    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:15.062866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:15.061361    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:15.819367 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:15.819280    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:16.172747 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:16.172630    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7107-729a69" event={"ID":"1c8bc351-3ae6-4189-bba2-f9d1b4230530","Type":"ContainerStarted","Data":"3a06a59e2fdf864e903f3971b6fbab65b6a7e64734d411ec6921a3ec00900b93"}
Nov 03 19:43:16.192130 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:16.192081    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7107-729a69" podStartSLOduration=1.719060582 podStartE2EDuration="4.192066481s" totalImagesPullingTime="2.473005899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:16.189066991 +0000 UTC m=+952.443374220" watchObservedRunningTime="2026-11-03 19:43:16.192066481 +0000 UTC m=+952.446373705"
Nov 03 19:43:16.542998 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:16.542969    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:16.979610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:16.979526    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7107-729a69"]
Nov 03 19:43:16.984255 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:16.984000    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7107-729a69"]
Nov 03 19:43:17.238418 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:17.238337    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:17.400723 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:17.400686    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:18.006646 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:18.006598    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:18.186011 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:18.185971    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7107-729a69" podUID="1c8bc351-3ae6-4189-bba2-f9d1b4230530" containerName="test-curl-7107-729a69" containerID="cri-o://3a06a59e2fdf864e903f3971b6fbab65b6a7e64734d411ec6921a3ec00900b93" gracePeriod=2
Nov 03 19:43:18.258322 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:18.258238    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:18.410286 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:18.410244    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:18.590862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:18.579640    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:18.898820 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:18.898727    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:19.058901 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:19.055020    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:19.927057 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:19.920887    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:20.402326 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:20.402293    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7107-729a69"
Nov 03 19:43:20.530943 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:20.529709    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8bc351-3ae6-4189-bba2-f9d1b4230530-kube-api-access-xppnv\" (UniqueName: \"kubernetes.io/projected/1c8bc351-3ae6-4189-bba2-f9d1b4230530-kube-api-access-xppnv\") pod \"1c8bc351-3ae6-4189-bba2-f9d1b4230530\" (UID: \"1c8bc351-3ae6-4189-bba2-f9d1b4230530\") "
Nov 03 19:43:20.552786 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:20.552744    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8bc351-3ae6-4189-bba2-f9d1b4230530-kube-api-access-xppnv" pod "1c8bc351-3ae6-4189-bba2-f9d1b4230530" (UID: "1c8bc351-3ae6-4189-bba2-f9d1b4230530"). InnerVolumeSpecName "kube-api-access-xppnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:20.632609 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:20.632564    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xppnv\" (UniqueName: \"kubernetes.io/projected/1c8bc351-3ae6-4189-bba2-f9d1b4230530-kube-api-access-xppnv\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:20.982859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:20.982803    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:21.231292 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:21.231253    5237 generic.go:356] "Generic (PLEG): container finished" podID="1c8bc351-3ae6-4189-bba2-f9d1b4230530" containerID="3a06a59e2fdf864e903f3971b6fbab65b6a7e64734d411ec6921a3ec00900b93" exitCode=137
Nov 03 19:43:21.231477 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:21.231299    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7107-729a69"
Nov 03 19:43:21.231477 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:21.231335    5237 scope.go:122] "RemoveContainer" containerID="3a06a59e2fdf864e903f3971b6fbab65b6a7e64734d411ec6921a3ec00900b93"
Nov 03 19:43:21.241959 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:21.241939    5237 scope.go:122] "RemoveContainer" containerID="3a06a59e2fdf864e903f3971b6fbab65b6a7e64734d411ec6921a3ec00900b93"
Nov 03 19:43:21.242255 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:43:21.242238    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a06a59e2fdf864e903f3971b6fbab65b6a7e64734d411ec6921a3ec00900b93\": container with ID starting with 3a06a59e2fdf864e903f3971b6fbab65b6a7e64734d411ec6921a3ec00900b93 not found: ID does not exist" containerID="3a06a59e2fdf864e903f3971b6fbab65b6a7e64734d411ec6921a3ec00900b93"
Nov 03 19:43:21.242303 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:21.242265    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a06a59e2fdf864e903f3971b6fbab65b6a7e64734d411ec6921a3ec00900b93"} err="failed to get container status \"3a06a59e2fdf864e903f3971b6fbab65b6a7e64734d411ec6921a3ec00900b93\": rpc error: code = NotFound desc = could not find container \"3a06a59e2fdf864e903f3971b6fbab65b6a7e64734d411ec6921a3ec00900b93\": container with ID starting with 3a06a59e2fdf864e903f3971b6fbab65b6a7e64734d411ec6921a3ec00900b93 not found: ID does not exist"
Nov 03 19:43:21.243423 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:43:21.243393    5237 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7107-729a69\" is forbidden: User \"system:node:ip-10-0-133-29.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-29.us-east-2.compute.internal' and this object" podUID="1c8bc351-3ae6-4189-bba2-f9d1b4230530" pod="opendatahub/test-curl-7107-729a69"
Nov 03 19:43:21.860674 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:21.860631    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c8bc351-3ae6-4189-bba2-f9d1b4230530" path="/var/lib/kubelet/pods/1c8bc351-3ae6-4189-bba2-f9d1b4230530/volumes"
Nov 03 19:43:22.107109 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:22.106310    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:22.302040 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:22.302002    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:22.517900 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:22.517861    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:22.850325 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:22.850291    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:23.086883 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:23.086800    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:23.620987 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:23.618546    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:24.601594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:24.595291    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:25.558658 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:25.558619    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:26.517630 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:26.515485    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:26.788443 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:26.788407    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:27.063098 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:27.063004    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:27.489471 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:27.489363    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:27.668823 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:27.668787    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:27.852275 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:27.852233    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:28.457658 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:28.457612    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:29.471213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:29.471180    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:30.152646 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.152611    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:30.304366 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.304326    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:30.358120 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.358088    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"]
Nov 03 19:43:30.358736 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.358716    5237 cpu_manager.go:414] "containerMap: removing container" podUID="1c8bc351-3ae6-4189-bba2-f9d1b4230530" containerName="test-curl-7107-729a69"
Nov 03 19:43:30.358856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.358738    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8bc351-3ae6-4189-bba2-f9d1b4230530" containerName="test-curl-7107-729a69"
Nov 03 19:43:30.358920 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.358890    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8bc351-3ae6-4189-bba2-f9d1b4230530" containerName="test-curl-7107-729a69"
Nov 03 19:43:30.363826 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.363802    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:43:30.378885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.378045    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"]
Nov 03 19:43:30.380527 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.380495    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ca12e9fb-serving-cert\""
Nov 03 19:43:30.468954 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.468858    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98833f3b-cfb4-4416-9429-18a25336a50b-maas-api-tls\") pod \"maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v\" (UID: \"98833f3b-cfb4-4416-9429-18a25336a50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:43:30.469126 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.468997    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98833f3b-cfb4-4416-9429-18a25336a50b-metrics-certs\") pod \"maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v\" (UID: \"98833f3b-cfb4-4416-9429-18a25336a50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:43:30.469194 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.469158    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbdvc\" (UniqueName: \"kubernetes.io/projected/98833f3b-cfb4-4416-9429-18a25336a50b-kube-api-access-rbdvc\") pod \"maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v\" (UID: \"98833f3b-cfb4-4416-9429-18a25336a50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:43:30.469256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.469235    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98833f3b-cfb4-4416-9429-18a25336a50b-openshift-service-ca\") pod \"maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v\" (UID: \"98833f3b-cfb4-4416-9429-18a25336a50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:43:30.520523 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.520442    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:30.571801 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.571761    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98833f3b-cfb4-4416-9429-18a25336a50b-openshift-service-ca\") pod \"maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v\" (UID: \"98833f3b-cfb4-4416-9429-18a25336a50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:43:30.573912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.573885    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98833f3b-cfb4-4416-9429-18a25336a50b-maas-api-tls\") pod \"maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v\" (UID: \"98833f3b-cfb4-4416-9429-18a25336a50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:43:30.574711 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.574689    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98833f3b-cfb4-4416-9429-18a25336a50b-metrics-certs\") pod \"maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v\" (UID: \"98833f3b-cfb4-4416-9429-18a25336a50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:43:30.575414 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.575391    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbdvc\" (UniqueName: \"kubernetes.io/projected/98833f3b-cfb4-4416-9429-18a25336a50b-kube-api-access-rbdvc\") pod \"maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v\" (UID: \"98833f3b-cfb4-4416-9429-18a25336a50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:43:30.576015 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.573678    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98833f3b-cfb4-4416-9429-18a25336a50b-openshift-service-ca\") pod \"maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v\" (UID: \"98833f3b-cfb4-4416-9429-18a25336a50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:43:30.592621 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.592589    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98833f3b-cfb4-4416-9429-18a25336a50b-maas-api-tls\") pod \"maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v\" (UID: \"98833f3b-cfb4-4416-9429-18a25336a50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:43:30.604964 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.604933    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98833f3b-cfb4-4416-9429-18a25336a50b-metrics-certs\") pod \"maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v\" (UID: \"98833f3b-cfb4-4416-9429-18a25336a50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:43:30.651857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.640246    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbdvc\" (UniqueName: \"kubernetes.io/projected/98833f3b-cfb4-4416-9429-18a25336a50b-kube-api-access-rbdvc\") pod \"maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v\" (UID: \"98833f3b-cfb4-4416-9429-18a25336a50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:43:30.700153 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.700116    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:43:30.771879 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:30.771062    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:31.116226 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:31.116192    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"]
Nov 03 19:43:31.116364 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:43:31.116326    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98833f3b_cfb4_4416_9429_18a25336a50b.slice/crio-4af6eb04a0a7ca3d0ac38b350f52a1b11c159044ad67334d86b8df87bee472e4 WatchSource:0}: Error finding container 4af6eb04a0a7ca3d0ac38b350f52a1b11c159044ad67334d86b8df87bee472e4: Status 404 returned error can't find the container with id 4af6eb04a0a7ca3d0ac38b350f52a1b11c159044ad67334d86b8df87bee472e4
Nov 03 19:43:31.533918 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:31.533804    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v" event={"ID":"98833f3b-cfb4-4416-9429-18a25336a50b","Type":"ContainerStarted","Data":"4af6eb04a0a7ca3d0ac38b350f52a1b11c159044ad67334d86b8df87bee472e4"}
Nov 03 19:43:31.871325 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:31.871286    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:32.544279 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:32.544240    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v" event={"ID":"98833f3b-cfb4-4416-9429-18a25336a50b","Type":"ContainerStarted","Data":"75799c5f0f3d9bf57a8b8e82f44ddd1ba36c7a508ba41ce798068ff5c831c33a"}
Nov 03 19:43:32.544766 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:32.544308    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:43:32.579191 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:32.579133    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v" podStartSLOduration=2.071970638 podStartE2EDuration="2.579118956s" totalImagesPullingTime="507.148318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:32.577627291 +0000 UTC m=+968.831934515" watchObservedRunningTime="2026-11-03 19:43:32.579118956 +0000 UTC m=+968.833426181"
Nov 03 19:43:32.585366 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:32.585324    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:32.790864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:32.786246    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:33.235861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:33.233353    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:35.168399 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:35.167947    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:35.249498 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:35.249464    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:35.358860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:35.357948    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:35.434922 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:35.426964    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:35.434922 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:35.432969    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:35.449890 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:35.449826    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:35.551178 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:35.551138    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:35.575318 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:35.575275    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:43:35.607192 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:35.607148    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:35.630559 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:35.630524    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:35.671109 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:35.671062    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:35.845077 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:35.845021    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:43:35.898685 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:35.862895    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 19:43:35.899768 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:35.862930    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 19:43:35.899768 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:35.882899    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:43:36.150698 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.132703    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 19:43:36.333868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.315597    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:43:36.365332 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.365280    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:36.379394 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.379355    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:43:36.385866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.382097    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:43:36.431545 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.429926    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:36.431545 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.430734    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:43:36.439612 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.439567    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:36.440366 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.440343    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:43:36.551668 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.551635    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:43:36.609867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.575051    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:36.609867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.577027    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:43:36.609867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.578014    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:36.609867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.578277    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:43:36.648528 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.621632    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:36.648528 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.621950    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:36.648528 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.622431    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:43:36.648528 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.645699    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:36.679485 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.677454    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:36.679485 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.677932    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:43:36.729671 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.729336    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:36.918662 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.918557    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 19:43:36.934055 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.919125    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:36.934055 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.919637    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:43:36.934055 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.919693    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:43:36.934055 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.921342    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:43:36.934055 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.921557    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:36.934055 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.921751    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:43:36.934055 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:36.921800    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:43:37.165884 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.141942    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 19:43:37.168312 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.168284    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:43:37.567868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.563278    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:43:37.672197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.583954    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:43:37.672197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.584260    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 19:43:37.672197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.583954    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:37.672197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.584982    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded"
Nov 03 19:43:37.672197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.614965    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 19:43:37.672197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.637946    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:37.672197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.659914    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:37.726141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.689076    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 19:43:37.726141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.725986    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:43:37.726141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.726054    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" podUID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:37.758726 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.758575    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:43:37.758726 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.758636    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" podUID="bd104015-ca9a-4193-bf07-f26376a66366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:37.928452 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.928336    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:37.929159 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.929121    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 19:43:37.930634 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.930600    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:37.931082 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:37.931059    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:43:38.075033 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.074986    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:38.216182 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.215991    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 19:43:38.247038 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.247002    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:38.568181 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.568138    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:38.569114 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.569082    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:43:38.587746 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.587706    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:43:38.588330 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.588300    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 19:43:38.615993 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.615938    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:38.616818 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.616791    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:43:38.623092 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.623062    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:43:38.634999 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.634955    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:38.649905 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.649862    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:38.663212 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.663169    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:38.778501 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.777898    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:38.851339 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.850233    5237 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:43:38.851339 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.850305    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" podUID="22e9f189-e173-4b40-b4a2-3455be004378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:38.981025 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.980964    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:38.984733 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.981304    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:38.985413 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.981335    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:38.986045 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:38.981377    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:39.259055 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:39.258955    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:39.625871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:39.599977    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:39.625871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:39.600571    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 19:43:39.625871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:39.600869    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:43:39.626692 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:39.626650    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:39.742863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:39.722577    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:39.742863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:39.722940    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:43:39.742863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:39.734709    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:43:39.796597 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:39.795985    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:40.052086 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:40.052024    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:40.078990 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:40.066365    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:40.078990 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:40.066757    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 19:43:40.078990 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:40.067058    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:43:40.203923 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:40.179956    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded"
Nov 03 19:43:40.273075 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:40.204226    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:40.301869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:40.273589    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:40.601877 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:40.601044    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:40.601877 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:40.601683    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:40.605858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:40.602809    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:40.630920 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:40.630538    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:40.695066 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:40.678145    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:43:40.729542 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:40.727527    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:40.729542 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:40.727658    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:40.729542 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:40.728029    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:43:40.748861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:40.737626    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:43:40.797061 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:40.796989    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 19:43:41.053173 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:41.052646    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:43:41.067247 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:41.067203    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:41.068938 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:41.067490    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:41.272865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:41.263130    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:41.350186 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:41.350085    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:41.402600 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:41.402565    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:43:41.446858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:41.441688    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:43:41.548878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:41.544205    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:43:41.619129 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:41.619004    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:41.627031 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:41.626992    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:41.631859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:41.631168    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:41.732381 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:41.732238    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:41.739562 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:41.732897    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:43:41.781385 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:41.781354    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:43:41.814586 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:41.809330    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:41.976872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:41.976753    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:42.091975 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:42.091928    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 19:43:42.352933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:42.351953    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 19:43:42.467865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:42.467156    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:43:42.482489 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:42.482452    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:42.488871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:42.487403    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:43:42.538928 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:42.533151    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:43:42.540992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:42.540963    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:43:42.542916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:42.542894    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:43:42.548997 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:42.548963    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:43:42.549949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:42.549906    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:43:42.554382 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:42.554360    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:43:42.648366 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:42.648269    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:42.910908 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:42.905518    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:43.260979 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:43:43.260881    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:43:44.028424 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:44.027054    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:44.869828 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:44.869787    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:45.254525 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:45.254424    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:45.464496 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:45.464453    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:45.621288 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:45.621247    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:45.903421 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:45.903331    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:46.251665 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.251575    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:46.545106 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.545066    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"]
Nov 03 19:43:46.550522 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.550495    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:43:46.559496 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.558896    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-edc935cf-serving-cert\""
Nov 03 19:43:46.563696 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.563672    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"]
Nov 03 19:43:46.632155 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.632111    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c90576f-d579-4d88-86da-2cc89afd446b-openshift-service-ca\") pod \"maas-api-e2e-ait-edc935cf-5599948fc7-58wc9\" (UID: \"6c90576f-d579-4d88-86da-2cc89afd446b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:43:46.632620 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.632586    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c90576f-d579-4d88-86da-2cc89afd446b-maas-api-tls\") pod \"maas-api-e2e-ait-edc935cf-5599948fc7-58wc9\" (UID: \"6c90576f-d579-4d88-86da-2cc89afd446b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:43:46.632727 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.632710    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlqhk\" (UniqueName: \"kubernetes.io/projected/6c90576f-d579-4d88-86da-2cc89afd446b-kube-api-access-mlqhk\") pod \"maas-api-e2e-ait-edc935cf-5599948fc7-58wc9\" (UID: \"6c90576f-d579-4d88-86da-2cc89afd446b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:43:46.632792 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.632777    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c90576f-d579-4d88-86da-2cc89afd446b-metrics-certs\") pod \"maas-api-e2e-ait-edc935cf-5599948fc7-58wc9\" (UID: \"6c90576f-d579-4d88-86da-2cc89afd446b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:43:46.776244 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.753793    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c90576f-d579-4d88-86da-2cc89afd446b-openshift-service-ca\") pod \"maas-api-e2e-ait-edc935cf-5599948fc7-58wc9\" (UID: \"6c90576f-d579-4d88-86da-2cc89afd446b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:43:46.776961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.776927    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c90576f-d579-4d88-86da-2cc89afd446b-maas-api-tls\") pod \"maas-api-e2e-ait-edc935cf-5599948fc7-58wc9\" (UID: \"6c90576f-d579-4d88-86da-2cc89afd446b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:43:46.777775 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.777750    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlqhk\" (UniqueName: \"kubernetes.io/projected/6c90576f-d579-4d88-86da-2cc89afd446b-kube-api-access-mlqhk\") pod \"maas-api-e2e-ait-edc935cf-5599948fc7-58wc9\" (UID: \"6c90576f-d579-4d88-86da-2cc89afd446b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:43:46.778346 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.778322    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c90576f-d579-4d88-86da-2cc89afd446b-metrics-certs\") pod \"maas-api-e2e-ait-edc935cf-5599948fc7-58wc9\" (UID: \"6c90576f-d579-4d88-86da-2cc89afd446b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:43:46.779020 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.770994    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c90576f-d579-4d88-86da-2cc89afd446b-openshift-service-ca\") pod \"maas-api-e2e-ait-edc935cf-5599948fc7-58wc9\" (UID: \"6c90576f-d579-4d88-86da-2cc89afd446b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:43:46.793203 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.793166    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:46.814874 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.814787    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c90576f-d579-4d88-86da-2cc89afd446b-metrics-certs\") pod \"maas-api-e2e-ait-edc935cf-5599948fc7-58wc9\" (UID: \"6c90576f-d579-4d88-86da-2cc89afd446b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:43:46.819918 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.816487    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c90576f-d579-4d88-86da-2cc89afd446b-maas-api-tls\") pod \"maas-api-e2e-ait-edc935cf-5599948fc7-58wc9\" (UID: \"6c90576f-d579-4d88-86da-2cc89afd446b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:43:46.827428 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.825778    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlqhk\" (UniqueName: \"kubernetes.io/projected/6c90576f-d579-4d88-86da-2cc89afd446b-kube-api-access-mlqhk\") pod \"maas-api-e2e-ait-edc935cf-5599948fc7-58wc9\" (UID: \"6c90576f-d579-4d88-86da-2cc89afd446b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:43:46.873153 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.871754    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:43:46.936336 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:46.936289    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:47.323856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:47.321397    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"]
Nov 03 19:43:47.399562 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:43:47.399167    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c90576f_d579_4d88_86da_2cc89afd446b.slice/crio-abfb96a0cded3651fc660450d440e0ed8c9eac1c31df4c241515bb3c32ec2e5e WatchSource:0}: Error finding container abfb96a0cded3651fc660450d440e0ed8c9eac1c31df4c241515bb3c32ec2e5e: Status 404 returned error can't find the container with id abfb96a0cded3651fc660450d440e0ed8c9eac1c31df4c241515bb3c32ec2e5e
Nov 03 19:43:47.552473 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:47.552427    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9" event={"ID":"6c90576f-d579-4d88-86da-2cc89afd446b","Type":"ContainerStarted","Data":"abfb96a0cded3651fc660450d440e0ed8c9eac1c31df4c241515bb3c32ec2e5e"}
Nov 03 19:43:47.755670 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:47.755590    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:48.560651 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:48.560618    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9" event={"ID":"6c90576f-d579-4d88-86da-2cc89afd446b","Type":"ContainerStarted","Data":"8ef719ec35f484edbefc5fc00c5a4f3f841cec87492ccf4b1d71f6492f7f0d20"}
Nov 03 19:43:48.561134 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:48.560710    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:43:48.595904 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:48.595805    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9" podStartSLOduration=2.06722947 podStartE2EDuration="2.595785322s" totalImagesPullingTime="528.555852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:48.595392235 +0000 UTC m=+984.849699461" watchObservedRunningTime="2026-11-03 19:43:48.595785322 +0000 UTC m=+984.850092560"
Nov 03 19:43:48.686061 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:48.686010    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:49.214227 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:49.214190    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:49.513275 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:49.513189    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:50.367552 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:50.366252    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:51.060819 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:51.060774    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:52.007715 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.007684    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:52.165266 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.165217    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:52.386027 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.385986    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"]
Nov 03 19:43:52.396794 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.396764    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.403508 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.403462    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-28ef3ba9-openshift-default-dockercfg-bhfrr\""
Nov 03 19:43:52.408633 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.408450    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"]
Nov 03 19:43:52.481747 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.481706    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:52.511212 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.511174    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-workload-socket\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.511436 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.511294    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-token\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.511436 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.511329    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26hlz\" (UniqueName: \"kubernetes.io/projected/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-kube-api-access-26hlz\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.511559 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.511448    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-envoy\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.511559 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.511544    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-podinfo\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.511668 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.511587    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-data\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.511668 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.511606    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-credential-socket\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.511779 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.511708    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-workload-certs\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.511859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.511782    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istiod-ca-cert\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.616344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.612995    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-workload-socket\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.616344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.613106    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-token\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.616344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.613152    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26hlz\" (UniqueName: \"kubernetes.io/projected/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-kube-api-access-26hlz\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.616344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.613194    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-envoy\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.616344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.613258    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-podinfo\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.616344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.613313    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-data\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.616344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.613341    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-credential-socket\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.616344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.613382    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-workload-certs\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.616344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.613449    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istiod-ca-cert\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.616344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.614433    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istiod-ca-cert\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.616344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.614778    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-workload-socket\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.616344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.615964    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-data\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.621343 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.621312    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-credential-socket\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.621778 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.621757    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-workload-certs\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.650561 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.650475    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:52.654654 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.654583    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-envoy\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.656692 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.655286    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-podinfo\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.664868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.660624    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26hlz\" (UniqueName: \"kubernetes.io/projected/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-kube-api-access-26hlz\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.666220 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.666195    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-token\") pod \"e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") " pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:52.719727 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:52.719687    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:53.252883 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:53.251063    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"]
Nov 03 19:43:53.252883 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:43:53.251781    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e7d6a3_de96_4066_a7dc_bd73a26da2c8.slice/crio-938ade248a052ac9cd7e72601ca5788b87545978ccd14734c1e0c0ad095a3e5b WatchSource:0}: Error finding container 938ade248a052ac9cd7e72601ca5788b87545978ccd14734c1e0c0ad095a3e5b: Status 404 returned error can't find the container with id 938ade248a052ac9cd7e72601ca5788b87545978ccd14734c1e0c0ad095a3e5b
Nov 03 19:43:53.255005 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:53.254971    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:43:53.255122 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:53.255048    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:43:53.255122 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:53.255077    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:43:53.661265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:53.661226    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s" event={"ID":"62e7d6a3-de96-4066-a7dc-bd73a26da2c8","Type":"ContainerStarted","Data":"0b4b262836f72ea6d04400a0e8e0f690142533bc30e2c0e708bad6eb99451cf8"}
Nov 03 19:43:53.661434 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:53.661297    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s" event={"ID":"62e7d6a3-de96-4066-a7dc-bd73a26da2c8","Type":"ContainerStarted","Data":"938ade248a052ac9cd7e72601ca5788b87545978ccd14734c1e0c0ad095a3e5b"}
Nov 03 19:43:53.692912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:53.692830    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s" podStartSLOduration=1.692815008 podStartE2EDuration="1.692815008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:53.689480684 +0000 UTC m=+989.943787910" watchObservedRunningTime="2026-11-03 19:43:53.692815008 +0000 UTC m=+989.947122301"
Nov 03 19:43:53.720367 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:53.720335    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:53.767246 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:53.767202    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:54.594639 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:54.594595    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:43:54.726518 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:54.726488    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:54.942940 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:54.935539    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:55.086526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:55.086489    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:55.296933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:55.296899    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:55.476693 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:55.476656    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:55.682250 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:55.682149    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:55.683176 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:55.683151    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:43:56.020166 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:56.020078    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:56.750012 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:56.749977    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:56.920103 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:56.920066    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:57.384758 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.379370    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"]
Nov 03 19:43:57.395134 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.389964    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.405339 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.405308    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-9b7a388c-openshift-default-dockercfg-2282w\""
Nov 03 19:43:57.435176 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.435133    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"]
Nov 03 19:43:57.635919 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.555240    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:57.635919 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.556651    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-podinfo\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.635919 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.635899    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-data\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.636255 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.635941    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-envoy\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.636255 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.636051    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-workload-socket\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.636255 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.636145    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-workload-certs\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.636587 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.636396    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crtls\" (UniqueName: \"kubernetes.io/projected/d39b8c6d-00c7-46b5-82a1-a531af3ba320-kube-api-access-crtls\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.645004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.636750    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-token\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.645004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.636805    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istiod-ca-cert\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.645004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.636872    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-credential-socket\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.788767 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.778480    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-data\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.788767 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.778544    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-envoy\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.788767 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.778658    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-workload-socket\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.788767 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.778742    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-workload-certs\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.788767 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.778780    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crtls\" (UniqueName: \"kubernetes.io/projected/d39b8c6d-00c7-46b5-82a1-a531af3ba320-kube-api-access-crtls\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.788767 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.778966    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-token\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.788767 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.779009    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istiod-ca-cert\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.788767 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.779055    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-credential-socket\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.788767 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.779163    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-podinfo\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.788767 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.787633    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-workload-certs\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.788767 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.787918    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-data\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.794177 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.794149    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istiod-ca-cert\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.794647 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.794623    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-credential-socket\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.796655 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.796632    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-workload-socket\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.846862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.845136    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-envoy\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.846862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.845646    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-podinfo\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.851187 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.850293    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:57.914904 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.914795    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-token\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:57.940628 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:57.940549    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crtls\" (UniqueName: \"kubernetes.io/projected/d39b8c6d-00c7-46b5-82a1-a531af3ba320-kube-api-access-crtls\") pod \"e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") " pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:58.035078 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:58.035036    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:58.178875 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:58.178270    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:58.419768 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:58.419493    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:58.425504 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:58.425470    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"]
Nov 03 19:43:58.428804 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:43:58.428777    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39b8c6d_00c7_46b5_82a1_a531af3ba320.slice/crio-02275ce096b4fac418d5b52959c29a4afa5bccb5d1e6831ebf18a41ab4e4a3cb WatchSource:0}: Error finding container 02275ce096b4fac418d5b52959c29a4afa5bccb5d1e6831ebf18a41ab4e4a3cb: Status 404 returned error can't find the container with id 02275ce096b4fac418d5b52959c29a4afa5bccb5d1e6831ebf18a41ab4e4a3cb
Nov 03 19:43:58.432207 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:58.432098    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:43:58.432207 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:58.432191    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:43:58.432387 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:58.432232    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:43:58.813875 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:58.813822    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd" event={"ID":"d39b8c6d-00c7-46b5-82a1-a531af3ba320","Type":"ContainerStarted","Data":"a61de96bf7654a462863b676a6af1e392d59ce1e9100ea1e2ac27584f50327e4"}
Nov 03 19:43:58.814381 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:58.813897    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd" event={"ID":"d39b8c6d-00c7-46b5-82a1-a531af3ba320","Type":"ContainerStarted","Data":"02275ce096b4fac418d5b52959c29a4afa5bccb5d1e6831ebf18a41ab4e4a3cb"}
Nov 03 19:43:59.036472 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:59.036427    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:59.041910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:59.041882    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:59.079444 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:59.079323    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd" podStartSLOduration=2.079299087 podStartE2EDuration="2.079299087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:58.846199196 +0000 UTC m=+995.100506421" watchObservedRunningTime="2026-11-03 19:43:59.079299087 +0000 UTC m=+995.333606315"
Nov 03 19:43:59.592405 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:59.592371    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:59.821010 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:59.820970    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:43:59.821975 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:43:59.821954    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:44:00.674299 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:00.674258    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:00.917897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:00.889336    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:01.581865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:01.571918    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:02.024248 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:02.024161    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:02.229868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:02.225012    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:03.119402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:03.119358    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:03.323105 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:03.323071    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:03.541934 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:03.539415    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:03.724861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:03.724779    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:04.315917 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:04.309484    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:04.485434 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:04.485403    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:05.179858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:05.178073    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:06.099055 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.098441    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:06.651704 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.651666    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"]
Nov 03 19:44:06.657459 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.657432    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:06.662903 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.662827    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-28ef3ba9-serving-cert\""
Nov 03 19:44:06.665081 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.665054    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"]
Nov 03 19:44:06.721399 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.721362    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:06.821866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.821094    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc24f10-b842-4e07-9da0-d176e93f9f27-openshift-service-ca\") pod \"maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f\" (UID: \"0fc24f10-b842-4e07-9da0-d176e93f9f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:06.821866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.821231    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fc24f10-b842-4e07-9da0-d176e93f9f27-maas-api-tls\") pod \"maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f\" (UID: \"0fc24f10-b842-4e07-9da0-d176e93f9f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:06.821866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.821438    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fc24f10-b842-4e07-9da0-d176e93f9f27-metrics-certs\") pod \"maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f\" (UID: \"0fc24f10-b842-4e07-9da0-d176e93f9f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:06.821866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.821508    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8798\" (UniqueName: \"kubernetes.io/projected/0fc24f10-b842-4e07-9da0-d176e93f9f27-kube-api-access-x8798\") pod \"maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f\" (UID: \"0fc24f10-b842-4e07-9da0-d176e93f9f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:06.867881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.867828    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:06.922985 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.922891    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fc24f10-b842-4e07-9da0-d176e93f9f27-metrics-certs\") pod \"maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f\" (UID: \"0fc24f10-b842-4e07-9da0-d176e93f9f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:06.922985 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.922956    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8798\" (UniqueName: \"kubernetes.io/projected/0fc24f10-b842-4e07-9da0-d176e93f9f27-kube-api-access-x8798\") pod \"maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f\" (UID: \"0fc24f10-b842-4e07-9da0-d176e93f9f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:06.923234 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.923074    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc24f10-b842-4e07-9da0-d176e93f9f27-openshift-service-ca\") pod \"maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f\" (UID: \"0fc24f10-b842-4e07-9da0-d176e93f9f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:06.923234 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.923096    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fc24f10-b842-4e07-9da0-d176e93f9f27-maas-api-tls\") pod \"maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f\" (UID: \"0fc24f10-b842-4e07-9da0-d176e93f9f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:06.923877 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.923828    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc24f10-b842-4e07-9da0-d176e93f9f27-openshift-service-ca\") pod \"maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f\" (UID: \"0fc24f10-b842-4e07-9da0-d176e93f9f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:06.925583 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.925556    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fc24f10-b842-4e07-9da0-d176e93f9f27-maas-api-tls\") pod \"maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f\" (UID: \"0fc24f10-b842-4e07-9da0-d176e93f9f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:06.925667 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.925634    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fc24f10-b842-4e07-9da0-d176e93f9f27-metrics-certs\") pod \"maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f\" (UID: \"0fc24f10-b842-4e07-9da0-d176e93f9f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:06.936044 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.936016    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8798\" (UniqueName: \"kubernetes.io/projected/0fc24f10-b842-4e07-9da0-d176e93f9f27-kube-api-access-x8798\") pod \"maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f\" (UID: \"0fc24f10-b842-4e07-9da0-d176e93f9f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:06.976181 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:06.976148    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:07.052359 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:07.052325    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:07.527903 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:07.527353    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"]
Nov 03 19:44:07.529894 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:44:07.529813    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc24f10_b842_4e07_9da0_d176e93f9f27.slice/crio-4ed4fe318a4076827d0355ead87d4eb64deb8b347d128c7810ac81a64791c585 WatchSource:0}: Error finding container 4ed4fe318a4076827d0355ead87d4eb64deb8b347d128c7810ac81a64791c585: Status 404 returned error can't find the container with id 4ed4fe318a4076827d0355ead87d4eb64deb8b347d128c7810ac81a64791c585
Nov 03 19:44:07.809166 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:07.809126    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:08.240159 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:08.240124    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f" event={"ID":"0fc24f10-b842-4e07-9da0-d176e93f9f27","Type":"ContainerStarted","Data":"d5c536aef5d20fcd07b2b3e573a08659e2a6dabfd6eeaf3fea9ba7d12442ccee"}
Nov 03 19:44:08.240330 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:08.240179    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f" event={"ID":"0fc24f10-b842-4e07-9da0-d176e93f9f27","Type":"ContainerStarted","Data":"4ed4fe318a4076827d0355ead87d4eb64deb8b347d128c7810ac81a64791c585"}
Nov 03 19:44:08.240330 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:08.240233    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:08.286854 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:08.286780    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f" podStartSLOduration=1.6779834390000001 podStartE2EDuration="2.286761273s" totalImagesPullingTime="608.777834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:08.285242795 +0000 UTC m=+1004.539550021" watchObservedRunningTime="2026-11-03 19:44:08.286761273 +0000 UTC m=+1004.541068502"
Nov 03 19:44:08.739754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:08.739716    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:08.902587 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:08.901927    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:09.522045 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:09.522006    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:10.240602 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.240485    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"]
Nov 03 19:44:10.275565 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.275534    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"]
Nov 03 19:44:10.275758 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.275653    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.347509 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.347472    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.347686 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.347635    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.347686 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.347669    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjpfs\" (UniqueName: \"kubernetes.io/projected/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-kube-api-access-rjpfs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.347757 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.347697    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.347816 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.347798    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.347886 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.347874    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.347932 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.347919    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.348007 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.347991    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.348055 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.348044    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.444806 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.444772    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"]
Nov 03 19:44:10.448920 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.448895    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:10.449732 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.449701    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.449881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.449775    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.449881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.449813    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.450010 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.449908    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.450063 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.450048    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.450167 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.450144    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.450208 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.450158    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.450245 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.450224    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjpfs\" (UniqueName: \"kubernetes.io/projected/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-kube-api-access-rjpfs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.450287 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.450255    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.450287 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.450279    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.450451 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.450430    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.451377 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.451293    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.451707 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.451682    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.451900 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.451685    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.454133 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.454101    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.454364 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.454348    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.461383 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.461364    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-9b7a388c-serving-cert\""
Nov 03 19:44:10.471739 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.471202    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjpfs\" (UniqueName: \"kubernetes.io/projected/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-kube-api-access-rjpfs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.471739 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.471571    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"]
Nov 03 19:44:10.474424 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.474160    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.552224 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.552185    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-maas-api-tls\") pod \"maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx\" (UID: \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:10.552466 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.552442    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-openshift-service-ca\") pod \"maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx\" (UID: \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:10.552588 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.552567    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2snb\" (UniqueName: \"kubernetes.io/projected/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-kube-api-access-q2snb\") pod \"maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx\" (UID: \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:10.552687 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.552674    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-metrics-certs\") pod \"maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx\" (UID: \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:10.563300 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.563257    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:10.594269 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.594229    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:10.665854 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.659728    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-openshift-service-ca\") pod \"maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx\" (UID: \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:10.665854 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.662941    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2snb\" (UniqueName: \"kubernetes.io/projected/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-kube-api-access-q2snb\") pod \"maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx\" (UID: \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:10.665854 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.663145    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-metrics-certs\") pod \"maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx\" (UID: \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:10.665854 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.663192    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-maas-api-tls\") pod \"maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx\" (UID: \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:10.665854 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.665088    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-openshift-service-ca\") pod \"maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx\" (UID: \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:10.696665 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.696635    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-maas-api-tls\") pod \"maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx\" (UID: \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:10.713954 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.703142    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2snb\" (UniqueName: \"kubernetes.io/projected/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-kube-api-access-q2snb\") pod \"maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx\" (UID: \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:10.743468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.734054    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-metrics-certs\") pod \"maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx\" (UID: \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:10.791278 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.791242    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:10.808231 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:10.808196    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:11.088451 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:11.088420    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"]
Nov 03 19:44:11.089559 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:44:11.089519    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33872d2d_bbf6_4242_a6c3_a0f3fffb54f7.slice/crio-f63840653d8a05432cd381f43d8479847e76dd3d24443206b7f974f3a932069f WatchSource:0}: Error finding container f63840653d8a05432cd381f43d8479847e76dd3d24443206b7f974f3a932069f: Status 404 returned error can't find the container with id f63840653d8a05432cd381f43d8479847e76dd3d24443206b7f974f3a932069f
Nov 03 19:44:11.092792 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:11.092754    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:44:11.092920 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:11.092858    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:44:11.092920 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:11.092913    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:44:11.162662 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:11.162276    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"]
Nov 03 19:44:11.293026 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:11.292986    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" event={"ID":"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7","Type":"ContainerStarted","Data":"49be4943cff4db862d760349b0104e187f0fae677588243f65ca07484e37f037"}
Nov 03 19:44:11.293598 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:11.293571    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" event={"ID":"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7","Type":"ContainerStarted","Data":"f63840653d8a05432cd381f43d8479847e76dd3d24443206b7f974f3a932069f"}
Nov 03 19:44:11.304398 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:11.304317    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx" event={"ID":"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2","Type":"ContainerStarted","Data":"450f7b6f1c7bb793cc3423f431149d660b8c6daf665733619844827ff3ce4c7c"}
Nov 03 19:44:11.357918 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:11.350627    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" podStartSLOduration=1.3506069809999999 podStartE2EDuration="1.350606981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:11.349230944 +0000 UTC m=+1007.603538172" watchObservedRunningTime="2026-11-03 19:44:11.350606981 +0000 UTC m=+1007.604914206"
Nov 03 19:44:11.600856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:11.596937    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:12.584540 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:12.584499    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:12.606715 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:12.604504    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:12.606715 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:12.605462    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:12.606715 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:12.605513    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" podUID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:44:12.616858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:12.613586    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:12.617135 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:12.617104    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:12.778062 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:12.777957    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:12.806058 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:12.806012    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:12.939992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:12.939941    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:12.940902 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:12.940252    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:13.002038 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.001984    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:13.242966 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.242923    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:13.259859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.255961    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:13.685212 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.685162    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:13.774860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.687181    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:44:13.776147 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.776115    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:44:13.827897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.759951    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 19:44:13.827897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.760025    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:44:13.827897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.776367    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:13.827897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.782075    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:44:13.827897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.792988    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:44:13.827897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.760000    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:44:13.827897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.793985    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:44:13.827897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.827133    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 19:44:13.827897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.827562    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:44:13.879872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.856946    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:13.879872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.857003    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" podUID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:44:13.994559 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.994458    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:13.995905 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:13.995874    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:14.025868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:14.015007    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:14.075439 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:14.075374    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:14.097868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:14.094982    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:14.284566 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:14.273822    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 19:44:14.493282 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:14.493245    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:14.497380 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:44:14.497344    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:44:14.526525 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:14.526454    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx" podStartSLOduration=3.85900519 podStartE2EDuration="4.52643135s" totalImagesPullingTime="667.42616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:14.523308869 +0000 UTC m=+1010.777616095" watchObservedRunningTime="2026-11-03 19:44:14.52643135 +0000 UTC m=+1010.780738576"
Nov 03 19:44:14.677577 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:14.675973    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:14.677577 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:14.676057    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" podUID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:44:14.770911 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:14.770866    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 19:44:14.833900 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:14.782286    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:44:14.833900 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:14.782466    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:14.892234 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:14.782555    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:44:14.927013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:14.926969    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:14.995920 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:14.975279    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:15.091261 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:44:14.981706    5237 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.096s"
Nov 03 19:44:15.091599 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.091562    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx" event={"ID":"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2","Type":"ContainerStarted","Data":"fb00570ca8460b28e87f7d5eb757164c35479cb28f1d89fe8bb746115c172f27"}
Nov 03 19:44:15.097097 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.097061    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:44:15.097380 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.097364    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:44:15.097553 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.097540    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:44:15.097708 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.097696    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:44:15.097925 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.097910    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:15.098115 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.098102    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:15.103098 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.103077    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:44:15.163412 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.163384    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:44:15.163740 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.163724    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:44:15.164360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.019443    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:15.177107 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.021619    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:44:15.178135 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.022896    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:44:15.196073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.023058    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:44:15.198478 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.027014    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:15.629880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.101273    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:15.629880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.198805    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:44:15.629880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.199173    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 19:44:15.629880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.276703    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:15.696007 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.640380    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:44:15.696007 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.603082    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:15.696007 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.640941    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" podUID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:44:15.870860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.868471    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 19:44:15.870860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.868919    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:44:15.870860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.869234    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:44:15.916805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.916712    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:16.123861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:16.112823    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:16.203916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:16.173588    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:44:16.203916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:16.185586    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:16.205850 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:16.205087    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:16.297890 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:15.946142    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:16.578878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:16.569980    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:44:16.578878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:16.571676    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:44:16.578878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:16.571752    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:44:16.631276 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:16.631241    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:16.631583 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:16.631548    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" podUID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:44:16.646876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:16.632287    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 19:44:16.647553 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:16.647506    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 19:44:16.707038 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:16.706992    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:17.004878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:16.975970    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:17.004878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:16.979074    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 19:44:17.004878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:16.987931    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:17.004878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:16.988294    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:44:17.004878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:16.988637    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:44:17.066867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.054422    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:17.122862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.121326    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 19:44:17.150083 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.148087    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:17.181934 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.173661    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:17.192632 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.192581    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:17.217648 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.217587    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:17.219488 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.218760    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:17.310705 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.309705    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:17.370073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.362956    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:17.462867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.459749    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:17.595644 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.595548    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:17.595921 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.595891    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" podUID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:44:17.649807 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.649759    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:17.665872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.659589    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:17.686777 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.685312    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"]
Nov 03 19:44:17.686777 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.685580    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v" podUID="98833f3b-cfb4-4416-9429-18a25336a50b" containerName="maas-api" containerID="cri-o://75799c5f0f3d9bf57a8b8e82f44ddd1ba36c7a508ba41ce798068ff5c831c33a" gracePeriod=30
Nov 03 19:44:18.001394 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.977648    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:44:18.001394 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:17.979716    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:18.003029 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.002386    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:44:18.003795 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.003503    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:44:18.004504 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.003965    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:44:18.149977 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.126503    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:18.149977 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.127022    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:18.206861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.198427    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:44:18.251064 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.229461    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:18.251064 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.246054    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:44:18.280237 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.260398    5237 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:44:18.288044 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.286958    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:18.371039 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.369829    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:18.390501 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.390438    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:44:18.399894 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.399155    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:18.497009 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.487191    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:18.613859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.597066    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:18.613859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.597143    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" podUID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:44:18.674446 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.674405    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v" podUID="98833f3b-cfb4-4416-9429-18a25336a50b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.70:8443/health\": dial tcp 10.133.0.70:8443: connect: connection refused"
Nov 03 19:44:18.674942 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.674900    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:18.679315 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.675147    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:18.679315 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.677310    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:44:18.958924 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:18.958876    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:44:19.115863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:19.000862    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:19.384939 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:19.119991    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:44:19.385674 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:19.385651    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:44:19.410937 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:19.196043    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:19.428794 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:19.224996    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 19:44:19.558738 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:19.558690    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:44:19.563751 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:19.563715    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 19:44:19.653966 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:19.646603    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:19.653966 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:19.646816    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:19.653966 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:19.646918    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" podUID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:44:19.669714 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:19.669678    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:44:19.670608 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:19.224942    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:19.679480 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:19.300268    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:19.776360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:19.692012    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 19:44:19.944739 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:19.944658    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:44:19.945090 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:19.945076    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:44:20.134059 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.126680    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:20.159328 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.159285    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:20.196130 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.195651    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:44:20.264941 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.257965    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:20.405362 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.405317    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:20.406924 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.406642    5237 generic.go:356] "Generic (PLEG): container finished" podID="98833f3b-cfb4-4416-9429-18a25336a50b" containerID="75799c5f0f3d9bf57a8b8e82f44ddd1ba36c7a508ba41ce798068ff5c831c33a" exitCode=0
Nov 03 19:44:20.593180 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.551277    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:44:20.593180 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.551455    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 19:44:20.593180 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.551560    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 19:44:20.593180 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.563184    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:20.593180 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.570060    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:20.684104 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.684059    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:20.759650 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.754807    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v" event={"ID":"98833f3b-cfb4-4416-9429-18a25336a50b","Type":"ContainerDied","Data":"75799c5f0f3d9bf57a8b8e82f44ddd1ba36c7a508ba41ce798068ff5c831c33a"}
Nov 03 19:44:20.826750 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.826716    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:44:20.827049 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.827031    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:44:20.827424 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.761162    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:20.827582 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.827560    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" podUID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:44:20.829691 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.829670    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:44:20.831638 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.831613    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:20.831867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.831661    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98833f3b-cfb4-4416-9429-18a25336a50b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/98833f3b-cfb4-4416-9429-18a25336a50b-metrics-certs\") pod \"98833f3b-cfb4-4416-9429-18a25336a50b\" (UID: \"98833f3b-cfb4-4416-9429-18a25336a50b\") "
Nov 03 19:44:20.832112 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.832096    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98833f3b-cfb4-4416-9429-18a25336a50b-kube-api-access-rbdvc\" (UniqueName: \"kubernetes.io/projected/98833f3b-cfb4-4416-9429-18a25336a50b-kube-api-access-rbdvc\") pod \"98833f3b-cfb4-4416-9429-18a25336a50b\" (UID: \"98833f3b-cfb4-4416-9429-18a25336a50b\") "
Nov 03 19:44:20.832357 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.832315    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:20.833099 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.833074    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98833f3b-cfb4-4416-9429-18a25336a50b-openshift-service-ca" pod "98833f3b-cfb4-4416-9429-18a25336a50b" (UID: "98833f3b-cfb4-4416-9429-18a25336a50b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:20.833237 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.832388    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98833f3b-cfb4-4416-9429-18a25336a50b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98833f3b-cfb4-4416-9429-18a25336a50b-openshift-service-ca\") pod \"98833f3b-cfb4-4416-9429-18a25336a50b\" (UID: \"98833f3b-cfb4-4416-9429-18a25336a50b\") "
Nov 03 19:44:20.833356 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.833341    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98833f3b-cfb4-4416-9429-18a25336a50b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98833f3b-cfb4-4416-9429-18a25336a50b-maas-api-tls\") pod \"98833f3b-cfb4-4416-9429-18a25336a50b\" (UID: \"98833f3b-cfb4-4416-9429-18a25336a50b\") "
Nov 03 19:44:20.833573 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.833552    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:44:20.834432 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.834409    5237 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98833f3b-cfb4-4416-9429-18a25336a50b-openshift-service-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:20.834860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.834813    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:20.898858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.896761    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:44:20.975029 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:20.974985    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98833f3b-cfb4-4416-9429-18a25336a50b-metrics-certs" pod "98833f3b-cfb4-4416-9429-18a25336a50b" (UID: "98833f3b-cfb4-4416-9429-18a25336a50b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:21.015607 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.002078    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98833f3b-cfb4-4416-9429-18a25336a50b-maas-api-tls" pod "98833f3b-cfb4-4416-9429-18a25336a50b" (UID: "98833f3b-cfb4-4416-9429-18a25336a50b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:21.035858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.017572    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98833f3b-cfb4-4416-9429-18a25336a50b-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:21.040964 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.040929    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98833f3b-cfb4-4416-9429-18a25336a50b-maas-api-tls\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:21.059047 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.059005    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98833f3b-cfb4-4416-9429-18a25336a50b-kube-api-access-rbdvc" pod "98833f3b-cfb4-4416-9429-18a25336a50b" (UID: "98833f3b-cfb4-4416-9429-18a25336a50b"). InnerVolumeSpecName "kube-api-access-rbdvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:21.065159 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.065108    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:44:21.068502 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.068474    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:21.098896 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.079934    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"]
Nov 03 19:44:21.098896 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.081024    5237 cpu_manager.go:414] "containerMap: removing container" podUID="98833f3b-cfb4-4416-9429-18a25336a50b" containerName="maas-api"
Nov 03 19:44:21.098896 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.081047    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98833f3b-cfb4-4416-9429-18a25336a50b" containerName="maas-api"
Nov 03 19:44:21.098896 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.081443    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="98833f3b-cfb4-4416-9429-18a25336a50b" containerName="maas-api"
Nov 03 19:44:21.100992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.100968    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.111882 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.111157    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-14a79569-openshift-default-dockercfg-k22gg\""
Nov 03 19:44:21.118868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.105726    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:44:21.159263 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.151961    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks788\" (UniqueName: \"kubernetes.io/projected/42eef399-4ea6-4e43-b54f-b00c9cce32f7-kube-api-access-ks788\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.159263 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.152011    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-data\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.159263 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.152141    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-workload-socket\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.159263 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.152221    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-podinfo\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.159263 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.152357    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istiod-ca-cert\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.164904 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.162996    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"]
Nov 03 19:44:21.165201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.165172    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-envoy\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.165371 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.165354    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-workload-certs\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.165440 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.165398    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-token\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.165614 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.165598    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-credential-socket\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.165732 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.165717    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbdvc\" (UniqueName: \"kubernetes.io/projected/98833f3b-cfb4-4416-9429-18a25336a50b-kube-api-access-rbdvc\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:21.248625 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.244867    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:44:21.249185 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.249155    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:21.466911 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.444514    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-credential-socket\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.467490 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.467451    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks788\" (UniqueName: \"kubernetes.io/projected/42eef399-4ea6-4e43-b54f-b00c9cce32f7-kube-api-access-ks788\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.486992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.486949    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-data\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.487314 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.487293    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-workload-socket\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.487473 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.487459    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-podinfo\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.487693 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.487676    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istiod-ca-cert\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.487892 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.487876    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-envoy\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.488153 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.488135    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-workload-certs\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.488267 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.488253    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-token\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.523310 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.499882    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-credential-socket\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.523563 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.454933    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:21.524063 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.524039    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-workload-certs\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.524219 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.519029    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istiod-ca-cert\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.524311 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.519595    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-workload-socket\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.524391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.518325    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-data\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.639859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.589181    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-envoy\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.678308 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.660959    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:21.678308 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.661019    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" podUID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:44:21.796914 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.723478    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-podinfo\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.796914 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.747867    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks788\" (UniqueName: \"kubernetes.io/projected/42eef399-4ea6-4e43-b54f-b00c9cce32f7-kube-api-access-ks788\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.924873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.892596    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-token\") pod \"e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") " pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:21.924873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.911925    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"
Nov 03 19:44:21.965999 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.965948    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v" event={"ID":"98833f3b-cfb4-4416-9429-18a25336a50b","Type":"ContainerDied","Data":"4af6eb04a0a7ca3d0ac38b350f52a1b11c159044ad67334d86b8df87bee472e4"}
Nov 03 19:44:21.965999 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:21.965997    5237 scope.go:122] "RemoveContainer" containerID="75799c5f0f3d9bf57a8b8e82f44ddd1ba36c7a508ba41ce798068ff5c831c33a"
Nov 03 19:44:22.012147 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:22.012116    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"]
Nov 03 19:44:22.017915 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:22.017886    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ca12e9fb-8546d56cb7-9ln9v"]
Nov 03 19:44:22.041121 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:44:22.041090    5237 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98833f3b_cfb4_4416_9429_18a25336a50b.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 19:44:22.121587 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:22.121502    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:22.422907 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:22.422437    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:22.559882 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:22.558925    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"]
Nov 03 19:44:22.565818 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:22.565772    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:44:22.566569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:22.566531    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:44:22.566685 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:22.566645    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:44:22.606380 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:22.606339    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:22.606633 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:22.606601    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" podUID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:44:22.932463 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:22.932408    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp" event={"ID":"42eef399-4ea6-4e43-b54f-b00c9cce32f7","Type":"ContainerStarted","Data":"0b5de6f9d3b4c5d73301acd6a584fe4fc7bff2d763f3cd4ef3a603280eab5c55"}
Nov 03 19:44:22.932647 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:22.932478    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp" event={"ID":"42eef399-4ea6-4e43-b54f-b00c9cce32f7","Type":"ContainerStarted","Data":"b207a6eb60b996080fb3509cb069e1898d09886052d335abd4b96b5ca5fc5ff3"}
Nov 03 19:44:22.960527 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:22.960397    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp" podStartSLOduration=1.960377896 podStartE2EDuration="1.960377896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:22.957645371 +0000 UTC m=+1019.211952598" watchObservedRunningTime="2026-11-03 19:44:22.960377896 +0000 UTC m=+1019.214685122"
Nov 03 19:44:23.135619 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:23.135586    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:23.158099 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:23.158047    5237 patch_prober.go:29] interesting pod/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:44:23.158406 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:23.158361    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp" podUID="42eef399-4ea6-4e43-b54f-b00c9cce32f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 19:44:23.595575 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:23.595539    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:23.596079 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:23.595598    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" podUID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:44:23.692859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:23.687185    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:23.863899 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:23.863785    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98833f3b-cfb4-4416-9429-18a25336a50b" path="/var/lib/kubelet/pods/98833f3b-cfb4-4416-9429-18a25336a50b/volumes"
Nov 03 19:44:24.128068 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:24.127995    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:24.626577 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:24.626540    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:24.654922 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:24.654888    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:24.655546 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:24.655515    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:24.657197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:24.657137    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:44:24.798296 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:24.798261    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:24.950279 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:24.950201    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:24.950960 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:24.950939    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:44:25.163200 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:25.163164    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:25.993164 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:25.993126    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:26.246917 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:26.206188    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:26.411912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:26.407120    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:27.460951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:27.454281    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:28.592904 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:28.545637    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:29.852858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:29.849547    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:30.208343 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:30.208247    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:30.605038 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:30.605007    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"]
Nov 03 19:44:30.605261 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:30.605215    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9" podUID="6c90576f-d579-4d88-86da-2cc89afd446b" containerName="maas-api" containerID="cri-o://8ef719ec35f484edbefc5fc00c5a4f3f841cec87492ccf4b1d71f6492f7f0d20" gracePeriod=30
Nov 03 19:44:30.711009 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:30.710971    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:30.960915 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:30.955056    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:31.231425 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.231335    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:31.240415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.240380    5237 generic.go:356] "Generic (PLEG): container finished" podID="6c90576f-d579-4d88-86da-2cc89afd446b" containerID="8ef719ec35f484edbefc5fc00c5a4f3f841cec87492ccf4b1d71f6492f7f0d20" exitCode=0
Nov 03 19:44:31.241305 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.240598    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9" event={"ID":"6c90576f-d579-4d88-86da-2cc89afd446b","Type":"ContainerDied","Data":"8ef719ec35f484edbefc5fc00c5a4f3f841cec87492ccf4b1d71f6492f7f0d20"}
Nov 03 19:44:31.254524 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.251344    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:44:31.466408 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.466188    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c90576f-d579-4d88-86da-2cc89afd446b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c90576f-d579-4d88-86da-2cc89afd446b-openshift-service-ca\") pod \"6c90576f-d579-4d88-86da-2cc89afd446b\" (UID: \"6c90576f-d579-4d88-86da-2cc89afd446b\") "
Nov 03 19:44:31.467087 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.467063    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c90576f-d579-4d88-86da-2cc89afd446b-kube-api-access-mlqhk\" (UniqueName: \"kubernetes.io/projected/6c90576f-d579-4d88-86da-2cc89afd446b-kube-api-access-mlqhk\") pod \"6c90576f-d579-4d88-86da-2cc89afd446b\" (UID: \"6c90576f-d579-4d88-86da-2cc89afd446b\") "
Nov 03 19:44:31.472051 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.467922    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c90576f-d579-4d88-86da-2cc89afd446b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c90576f-d579-4d88-86da-2cc89afd446b-metrics-certs\") pod \"6c90576f-d579-4d88-86da-2cc89afd446b\" (UID: \"6c90576f-d579-4d88-86da-2cc89afd446b\") "
Nov 03 19:44:31.474363 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.474342    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c90576f-d579-4d88-86da-2cc89afd446b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c90576f-d579-4d88-86da-2cc89afd446b-maas-api-tls\") pod \"6c90576f-d579-4d88-86da-2cc89afd446b\" (UID: \"6c90576f-d579-4d88-86da-2cc89afd446b\") "
Nov 03 19:44:31.477885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.467368    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c90576f-d579-4d88-86da-2cc89afd446b-openshift-service-ca" pod "6c90576f-d579-4d88-86da-2cc89afd446b" (UID: "6c90576f-d579-4d88-86da-2cc89afd446b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:31.489745 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.486599    5237 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c90576f-d579-4d88-86da-2cc89afd446b-openshift-service-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:31.491216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.491120    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:31.491986 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.491819    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c90576f-d579-4d88-86da-2cc89afd446b-kube-api-access-mlqhk" pod "6c90576f-d579-4d88-86da-2cc89afd446b" (UID: "6c90576f-d579-4d88-86da-2cc89afd446b"). InnerVolumeSpecName "kube-api-access-mlqhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:31.496680 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.495068    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c90576f-d579-4d88-86da-2cc89afd446b-metrics-certs" pod "6c90576f-d579-4d88-86da-2cc89afd446b" (UID: "6c90576f-d579-4d88-86da-2cc89afd446b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:31.496680 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.495125    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c90576f-d579-4d88-86da-2cc89afd446b-maas-api-tls" pod "6c90576f-d579-4d88-86da-2cc89afd446b" (UID: "6c90576f-d579-4d88-86da-2cc89afd446b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:31.684861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.684536    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlqhk\" (UniqueName: \"kubernetes.io/projected/6c90576f-d579-4d88-86da-2cc89afd446b-kube-api-access-mlqhk\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:31.684861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.684580    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c90576f-d579-4d88-86da-2cc89afd446b-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:31.684861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.684592    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c90576f-d579-4d88-86da-2cc89afd446b-maas-api-tls\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:31.719236 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:31.719148    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:32.252191 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:32.252121    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"
Nov 03 19:44:32.252615 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:32.252122    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9" event={"ID":"6c90576f-d579-4d88-86da-2cc89afd446b","Type":"ContainerDied","Data":"abfb96a0cded3651fc660450d440e0ed8c9eac1c31df4c241515bb3c32ec2e5e"}
Nov 03 19:44:32.252615 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:32.252251    5237 scope.go:122] "RemoveContainer" containerID="8ef719ec35f484edbefc5fc00c5a4f3f841cec87492ccf4b1d71f6492f7f0d20"
Nov 03 19:44:32.294365 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:32.294333    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"]
Nov 03 19:44:32.304804 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:32.304776    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc935cf-5599948fc7-58wc9"]
Nov 03 19:44:32.620704 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:32.620665    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:32.817712 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:32.817654    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:33.317828 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:33.317791    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:33.860669 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:33.860638    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c90576f-d579-4d88-86da-2cc89afd446b" path="/var/lib/kubelet/pods/6c90576f-d579-4d88-86da-2cc89afd446b/volumes"
Nov 03 19:44:34.231659 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:34.231569    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:35.131858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.122428    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:35.630844 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.630793    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"]
Nov 03 19:44:35.631443 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.631426    5237 cpu_manager.go:414] "containerMap: removing container" podUID="6c90576f-d579-4d88-86da-2cc89afd446b" containerName="maas-api"
Nov 03 19:44:35.631522 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.631444    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c90576f-d579-4d88-86da-2cc89afd446b" containerName="maas-api"
Nov 03 19:44:35.631568 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.631552    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c90576f-d579-4d88-86da-2cc89afd446b" containerName="maas-api"
Nov 03 19:44:35.636391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.636371    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:44:35.649753 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.649723    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"]
Nov 03 19:44:35.652652 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.652622    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:35.667901 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.667871    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-14a79569-serving-cert\""
Nov 03 19:44:35.794360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.791253    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97v5l\" (UniqueName: \"kubernetes.io/projected/aa10dff6-a42a-4d78-a4e0-c6828e772701-kube-api-access-97v5l\") pod \"maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x\" (UID: \"aa10dff6-a42a-4d78-a4e0-c6828e772701\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:44:35.794360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.791404    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa10dff6-a42a-4d78-a4e0-c6828e772701-maas-api-tls\") pod \"maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x\" (UID: \"aa10dff6-a42a-4d78-a4e0-c6828e772701\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:44:35.794360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.791512    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa10dff6-a42a-4d78-a4e0-c6828e772701-metrics-certs\") pod \"maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x\" (UID: \"aa10dff6-a42a-4d78-a4e0-c6828e772701\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:44:35.794360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.791617    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa10dff6-a42a-4d78-a4e0-c6828e772701-openshift-service-ca\") pod \"maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x\" (UID: \"aa10dff6-a42a-4d78-a4e0-c6828e772701\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:44:35.816442 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.816404    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:35.897339 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.897250    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa10dff6-a42a-4d78-a4e0-c6828e772701-openshift-service-ca\") pod \"maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x\" (UID: \"aa10dff6-a42a-4d78-a4e0-c6828e772701\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:44:35.897748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.897691    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97v5l\" (UniqueName: \"kubernetes.io/projected/aa10dff6-a42a-4d78-a4e0-c6828e772701-kube-api-access-97v5l\") pod \"maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x\" (UID: \"aa10dff6-a42a-4d78-a4e0-c6828e772701\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:44:35.898013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.897993    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa10dff6-a42a-4d78-a4e0-c6828e772701-maas-api-tls\") pod \"maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x\" (UID: \"aa10dff6-a42a-4d78-a4e0-c6828e772701\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:44:35.898211 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.898196    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa10dff6-a42a-4d78-a4e0-c6828e772701-metrics-certs\") pod \"maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x\" (UID: \"aa10dff6-a42a-4d78-a4e0-c6828e772701\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:44:35.900810 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.900786    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa10dff6-a42a-4d78-a4e0-c6828e772701-openshift-service-ca\") pod \"maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x\" (UID: \"aa10dff6-a42a-4d78-a4e0-c6828e772701\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:44:35.904306 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.904283    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa10dff6-a42a-4d78-a4e0-c6828e772701-maas-api-tls\") pod \"maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x\" (UID: \"aa10dff6-a42a-4d78-a4e0-c6828e772701\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:44:35.905317 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.905298    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa10dff6-a42a-4d78-a4e0-c6828e772701-metrics-certs\") pod \"maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x\" (UID: \"aa10dff6-a42a-4d78-a4e0-c6828e772701\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:44:35.920212 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.920183    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97v5l\" (UniqueName: \"kubernetes.io/projected/aa10dff6-a42a-4d78-a4e0-c6828e772701-kube-api-access-97v5l\") pod \"maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x\" (UID: \"aa10dff6-a42a-4d78-a4e0-c6828e772701\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:44:35.952487 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:35.952456    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:44:36.254991 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:44:36.254952    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa10dff6_a42a_4d78_a4e0_c6828e772701.slice/crio-147966900cc10d5f6443a500ba100859f3502369f5b0cca112f6b397e22f2224 WatchSource:0}: Error finding container 147966900cc10d5f6443a500ba100859f3502369f5b0cca112f6b397e22f2224: Status 404 returned error can't find the container with id 147966900cc10d5f6443a500ba100859f3502369f5b0cca112f6b397e22f2224
Nov 03 19:44:36.264188 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:36.264157    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"]
Nov 03 19:44:36.421859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:36.415429    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x" event={"ID":"aa10dff6-a42a-4d78-a4e0-c6828e772701","Type":"ContainerStarted","Data":"147966900cc10d5f6443a500ba100859f3502369f5b0cca112f6b397e22f2224"}
Nov 03 19:44:36.994862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:36.987388    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:37.423932 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:37.423887    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x" event={"ID":"aa10dff6-a42a-4d78-a4e0-c6828e772701","Type":"ContainerStarted","Data":"fc2fb436858a0066314e02e3792d5ad52aba1ae3d765c5e2020a58fb4d453664"}
Nov 03 19:44:37.424316 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:37.424098    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:44:37.445032 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:37.444972    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x" podStartSLOduration=1.9079205959999999 podStartE2EDuration="2.444957739s" totalImagesPullingTime="537.037143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:37.443968573 +0000 UTC m=+1033.698275797" watchObservedRunningTime="2026-11-03 19:44:37.444957739 +0000 UTC m=+1033.699264964"
Nov 03 19:44:38.135426 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:38.135384    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:38.881266 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:38.881219    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:39.614099 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:39.614063    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:40.292281 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:40.292241    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:41.012081 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:41.012025    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:41.719695 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:41.719657    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:42.236376 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:42.236344    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:42.370089 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:42.370055    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:42.522924 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:42.521952    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:43.141078 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:43.141043    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:43.297470 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:44:43.297436    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:44:43.437704 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:43.437625    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:44:43.935859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:43.928821    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:44.522778 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:44.522738    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:45.486605 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:45.486570    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:46.308457 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:46.308422    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:46.525937 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:46.523759    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:47.417165 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:47.417125    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:47.896861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:47.896732    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:48.038522 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:48.038485    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:48.157951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:48.150540    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"]
Nov 03 19:44:48.157951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:48.150857    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f" podUID="0fc24f10-b842-4e07-9da0-d176e93f9f27" containerName="maas-api" containerID="cri-o://d5c536aef5d20fcd07b2b3e573a08659e2a6dabfd6eeaf3fea9ba7d12442ccee" gracePeriod=30
Nov 03 19:44:48.234129 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:48.234097    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:48.399873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:48.393471    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:48.679093 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:48.679052    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:48.808990 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:48.808947    5237 generic.go:356] "Generic (PLEG): container finished" podID="0fc24f10-b842-4e07-9da0-d176e93f9f27" containerID="d5c536aef5d20fcd07b2b3e573a08659e2a6dabfd6eeaf3fea9ba7d12442ccee" exitCode=0
Nov 03 19:44:48.809172 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:48.809056    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f" event={"ID":"0fc24f10-b842-4e07-9da0-d176e93f9f27","Type":"ContainerDied","Data":"d5c536aef5d20fcd07b2b3e573a08659e2a6dabfd6eeaf3fea9ba7d12442ccee"}
Nov 03 19:44:48.909891 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:48.905956    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:49.097942 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.092451    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:49.170873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.163447    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc24f10-b842-4e07-9da0-d176e93f9f27-kube-api-access-x8798\" (UniqueName: \"kubernetes.io/projected/0fc24f10-b842-4e07-9da0-d176e93f9f27-kube-api-access-x8798\") pod \"0fc24f10-b842-4e07-9da0-d176e93f9f27\" (UID: \"0fc24f10-b842-4e07-9da0-d176e93f9f27\") "
Nov 03 19:44:49.170873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.163593    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc24f10-b842-4e07-9da0-d176e93f9f27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc24f10-b842-4e07-9da0-d176e93f9f27-openshift-service-ca\") pod \"0fc24f10-b842-4e07-9da0-d176e93f9f27\" (UID: \"0fc24f10-b842-4e07-9da0-d176e93f9f27\") "
Nov 03 19:44:49.170873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.163628    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fc24f10-b842-4e07-9da0-d176e93f9f27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fc24f10-b842-4e07-9da0-d176e93f9f27-maas-api-tls\") pod \"0fc24f10-b842-4e07-9da0-d176e93f9f27\" (UID: \"0fc24f10-b842-4e07-9da0-d176e93f9f27\") "
Nov 03 19:44:49.170873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.163748    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fc24f10-b842-4e07-9da0-d176e93f9f27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fc24f10-b842-4e07-9da0-d176e93f9f27-metrics-certs\") pod \"0fc24f10-b842-4e07-9da0-d176e93f9f27\" (UID: \"0fc24f10-b842-4e07-9da0-d176e93f9f27\") "
Nov 03 19:44:49.207393 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.192792    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc24f10-b842-4e07-9da0-d176e93f9f27-openshift-service-ca" pod "0fc24f10-b842-4e07-9da0-d176e93f9f27" (UID: "0fc24f10-b842-4e07-9da0-d176e93f9f27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:49.245945 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.245350    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"]
Nov 03 19:44:49.245945 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.245610    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s" podUID="62e7d6a3-de96-4066-a7dc-bd73a26da2c8" containerName="istio-proxy" containerID="cri-o://0b4b262836f72ea6d04400a0e8e0f690142533bc30e2c0e708bad6eb99451cf8" gracePeriod=30
Nov 03 19:44:49.258829 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.258792    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:49.279859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.266728    5237 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc24f10-b842-4e07-9da0-d176e93f9f27-openshift-service-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:49.394775 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.394692    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc24f10-b842-4e07-9da0-d176e93f9f27-kube-api-access-x8798" pod "0fc24f10-b842-4e07-9da0-d176e93f9f27" (UID: "0fc24f10-b842-4e07-9da0-d176e93f9f27"). InnerVolumeSpecName "kube-api-access-x8798". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:49.434368 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.433967    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc24f10-b842-4e07-9da0-d176e93f9f27-metrics-certs" pod "0fc24f10-b842-4e07-9da0-d176e93f9f27" (UID: "0fc24f10-b842-4e07-9da0-d176e93f9f27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:49.434780 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.434737    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc24f10-b842-4e07-9da0-d176e93f9f27-maas-api-tls" pod "0fc24f10-b842-4e07-9da0-d176e93f9f27" (UID: "0fc24f10-b842-4e07-9da0-d176e93f9f27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:49.495475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.495356    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fc24f10-b842-4e07-9da0-d176e93f9f27-maas-api-tls\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:49.495475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.495392    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fc24f10-b842-4e07-9da0-d176e93f9f27-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:49.495475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.495417    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8798\" (UniqueName: \"kubernetes.io/projected/0fc24f10-b842-4e07-9da0-d176e93f9f27-kube-api-access-x8798\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:49.909302 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.900974    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f" event={"ID":"0fc24f10-b842-4e07-9da0-d176e93f9f27","Type":"ContainerDied","Data":"4ed4fe318a4076827d0355ead87d4eb64deb8b347d128c7810ac81a64791c585"}
Nov 03 19:44:49.909302 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.901068    5237 scope.go:122] "RemoveContainer" containerID="d5c536aef5d20fcd07b2b3e573a08659e2a6dabfd6eeaf3fea9ba7d12442ccee"
Nov 03 19:44:49.909302 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.901368    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"
Nov 03 19:44:49.951597 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.951553    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"]
Nov 03 19:44:49.957941 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:49.957908    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28ef3ba9-5dcc466655-gbd7f"]
Nov 03 19:44:50.140081 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:50.140042    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:50.364938 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:50.364896    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:50.522473 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:50.522441    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:50.683909 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:50.680041    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:51.262059 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:51.261952    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"]
Nov 03 19:44:51.262558 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:51.262160    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx" podUID="f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2" containerName="maas-api" containerID="cri-o://fb00570ca8460b28e87f7d5eb757164c35479cb28f1d89fe8bb746115c172f27" gracePeriod=30
Nov 03 19:44:51.387324 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:51.387238    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:52.000864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:51.995915    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc24f10-b842-4e07-9da0-d176e93f9f27" path="/var/lib/kubelet/pods/0fc24f10-b842-4e07-9da0-d176e93f9f27/volumes"
Nov 03 19:44:52.013530 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:52.013494    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:52.015403 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:52.015374    5237 generic.go:356] "Generic (PLEG): container finished" podID="f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2" containerID="fb00570ca8460b28e87f7d5eb757164c35479cb28f1d89fe8bb746115c172f27" exitCode=0
Nov 03 19:44:52.015743 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:52.015723    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx" event={"ID":"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2","Type":"ContainerDied","Data":"fb00570ca8460b28e87f7d5eb757164c35479cb28f1d89fe8bb746115c172f27"}
Nov 03 19:44:52.073359 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:52.073315    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx" podUID="f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.76:8443/health\": dial tcp 10.133.0.76:8443: connect: connection refused"
Nov 03 19:44:52.418039 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:52.417820    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:52.720062 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:52.715988    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:52.908862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:52.881645    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:52.908862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:52.905943    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:52.922756 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:52.922718    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:52.977944 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:52.964926    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:52.999552 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:52.999516    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"]
Nov 03 19:44:53.004063 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.004004    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd" podUID="d39b8c6d-00c7-46b5-82a1-a531af3ba320" containerName="istio-proxy" containerID="cri-o://a61de96bf7654a462863b676a6af1e392d59ce1e9100ea1e2ac27584f50327e4" gracePeriod=30
Nov 03 19:44:53.128881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.116510    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:53.128881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.128604    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:53.234439 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.232478    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:53.263096 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.263042    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:53.264710 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.264054    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:53.329442 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.327945    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 19:44:53.409890 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.407240    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-openshift-service-ca\") pod \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\" (UID: \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\") "
Nov 03 19:44:53.409890 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.407376    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-metrics-certs\") pod \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\" (UID: \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\") "
Nov 03 19:44:53.409890 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.407590    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-maas-api-tls\") pod \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\" (UID: \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\") "
Nov 03 19:44:53.409890 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.407668    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-kube-api-access-q2snb\" (UniqueName: \"kubernetes.io/projected/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-kube-api-access-q2snb\") pod \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\" (UID: \"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2\") "
Nov 03 19:44:53.455888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.455042    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-openshift-service-ca" pod "f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2" (UID: "f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:53.563729 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.563673    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-kube-api-access-q2snb" pod "f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2" (UID: "f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2"). InnerVolumeSpecName "kube-api-access-q2snb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:53.564119 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.564092    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-maas-api-tls" pod "f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2" (UID: "f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:53.564343 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.564324    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-metrics-certs" pod "f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2" (UID: "f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:53.607918 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.601374    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:53.608481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.608448    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-maas-api-tls\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:53.608902 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.608879    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2snb\" (UniqueName: \"kubernetes.io/projected/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-kube-api-access-q2snb\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:53.609088 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.609072    5237 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2-openshift-service-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:53.726861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.724221    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:53.775273 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.775203    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:53.867017 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.866918    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:53.972867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.965205    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:54.004859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:53.981516    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:54.014871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.012249    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:54.026229 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.026193    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:44:54.026546 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.026528    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:44:54.121539 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.121446    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 19:44:54.124448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.124421    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:44:54.165066 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.161076    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:44:54.236046 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.235638    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:54.236236 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.236076    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:44:54.236236 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.236195    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:44:54.318548 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.318506    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:54.323727 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.323703    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"
Nov 03 19:44:54.324620 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.324599    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx" event={"ID":"f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2","Type":"ContainerDied","Data":"450f7b6f1c7bb793cc3423f431149d660b8c6daf665733619844827ff3ce4c7c"}
Nov 03 19:44:54.324789 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.324772    5237 scope.go:122] "RemoveContainer" containerID="fb00570ca8460b28e87f7d5eb757164c35479cb28f1d89fe8bb746115c172f27"
Nov 03 19:44:54.331465 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.331447    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:44:54.352218 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.352191    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:44:54.530817 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.530783    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"]
Nov 03 19:44:54.543455 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.543413    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b7a388c-68db448dbf-gnrcx"]
Nov 03 19:44:54.834857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.834820    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:44:54.941356 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.941313    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-envoy\") pod \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") "
Nov 03 19:44:54.941564 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.941418    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-podinfo\") pod \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") "
Nov 03 19:44:54.941564 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.941453    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-credential-socket\") pod \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") "
Nov 03 19:44:54.941564 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.941542    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-workload-socket\") pod \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") "
Nov 03 19:44:54.941745 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.941634    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-kube-api-access-26hlz\" (UniqueName: \"kubernetes.io/projected/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-kube-api-access-26hlz\") pod \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") "
Nov 03 19:44:54.941803 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.941767    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istiod-ca-cert\") pod \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") "
Nov 03 19:44:54.959857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.941933    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-workload-certs\") pod \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") "
Nov 03 19:44:54.959857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.941990    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-data\") pod \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") "
Nov 03 19:44:54.959857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.942027    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-token\" (UniqueName: \"kubernetes.io/projected/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-token\") pod \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\" (UID: \"62e7d6a3-de96-4066-a7dc-bd73a26da2c8\") "
Nov 03 19:44:54.959857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.943770    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:54.959857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.944250    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-workload-socket" pod "62e7d6a3-de96-4066-a7dc-bd73a26da2c8" (UID: "62e7d6a3-de96-4066-a7dc-bd73a26da2c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:44:54.959857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.945243    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istiod-ca-cert" pod "62e7d6a3-de96-4066-a7dc-bd73a26da2c8" (UID: "62e7d6a3-de96-4066-a7dc-bd73a26da2c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:54.959857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.946536    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-data" pod "62e7d6a3-de96-4066-a7dc-bd73a26da2c8" (UID: "62e7d6a3-de96-4066-a7dc-bd73a26da2c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:44:54.959857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.946752    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-credential-socket" pod "62e7d6a3-de96-4066-a7dc-bd73a26da2c8" (UID: "62e7d6a3-de96-4066-a7dc-bd73a26da2c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:44:54.959857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.947398    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-workload-certs" pod "62e7d6a3-de96-4066-a7dc-bd73a26da2c8" (UID: "62e7d6a3-de96-4066-a7dc-bd73a26da2c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:44:54.987226 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.978039    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-kube-api-access-26hlz" pod "62e7d6a3-de96-4066-a7dc-bd73a26da2c8" (UID: "62e7d6a3-de96-4066-a7dc-bd73a26da2c8"). InnerVolumeSpecName "kube-api-access-26hlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:54.988167 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.987577    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-token" pod "62e7d6a3-de96-4066-a7dc-bd73a26da2c8" (UID: "62e7d6a3-de96-4066-a7dc-bd73a26da2c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:54.988167 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:54.987667    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-envoy" pod "62e7d6a3-de96-4066-a7dc-bd73a26da2c8" (UID: "62e7d6a3-de96-4066-a7dc-bd73a26da2c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:44:55.023869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.019982    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-podinfo" pod "62e7d6a3-de96-4066-a7dc-bd73a26da2c8" (UID: "62e7d6a3-de96-4066-a7dc-bd73a26da2c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:44:55.071238 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.071191    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:55.071456 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.071436    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26hlz\" (UniqueName: \"kubernetes.io/projected/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-kube-api-access-26hlz\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:55.071591 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.071576    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:55.071707 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.071693    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:55.071807 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.071795    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:55.071935 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.071919    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:55.072051 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.072038    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:55.076234 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.072148    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:55.076446 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.076424    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e7d6a3-de96-4066-a7dc-bd73a26da2c8-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:55.330324 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.330230    5237 generic.go:356] "Generic (PLEG): container finished" podID="62e7d6a3-de96-4066-a7dc-bd73a26da2c8" containerID="0b4b262836f72ea6d04400a0e8e0f690142533bc30e2c0e708bad6eb99451cf8" exitCode=0
Nov 03 19:44:55.330324 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.330282    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"
Nov 03 19:44:55.330324 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.330312    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s" event={"ID":"62e7d6a3-de96-4066-a7dc-bd73a26da2c8","Type":"ContainerDied","Data":"0b4b262836f72ea6d04400a0e8e0f690142533bc30e2c0e708bad6eb99451cf8"}
Nov 03 19:44:55.330664 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.330376    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s" event={"ID":"62e7d6a3-de96-4066-a7dc-bd73a26da2c8","Type":"ContainerDied","Data":"938ade248a052ac9cd7e72601ca5788b87545978ccd14734c1e0c0ad095a3e5b"}
Nov 03 19:44:55.330664 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.330394    5237 scope.go:122] "RemoveContainer" containerID="0b4b262836f72ea6d04400a0e8e0f690142533bc30e2c0e708bad6eb99451cf8"
Nov 03 19:44:55.342238 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.342170    5237 scope.go:122] "RemoveContainer" containerID="0b4b262836f72ea6d04400a0e8e0f690142533bc30e2c0e708bad6eb99451cf8"
Nov 03 19:44:55.342577 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:44:55.342552    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4b262836f72ea6d04400a0e8e0f690142533bc30e2c0e708bad6eb99451cf8\": container with ID starting with 0b4b262836f72ea6d04400a0e8e0f690142533bc30e2c0e708bad6eb99451cf8 not found: ID does not exist" containerID="0b4b262836f72ea6d04400a0e8e0f690142533bc30e2c0e708bad6eb99451cf8"
Nov 03 19:44:55.342658 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.342583    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4b262836f72ea6d04400a0e8e0f690142533bc30e2c0e708bad6eb99451cf8"} err="failed to get container status \"0b4b262836f72ea6d04400a0e8e0f690142533bc30e2c0e708bad6eb99451cf8\": rpc error: code = NotFound desc = could not find container \"0b4b262836f72ea6d04400a0e8e0f690142533bc30e2c0e708bad6eb99451cf8\": container with ID starting with 0b4b262836f72ea6d04400a0e8e0f690142533bc30e2c0e708bad6eb99451cf8 not found: ID does not exist"
Nov 03 19:44:55.358666 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.358629    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"]
Nov 03 19:44:55.364893 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.364862    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-28ef3ba9-openshift-default-7bbf7bf4d5-x422s"]
Nov 03 19:44:55.791876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.788890    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:55.907876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.893203    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62e7d6a3-de96-4066-a7dc-bd73a26da2c8" path="/var/lib/kubelet/pods/62e7d6a3-de96-4066-a7dc-bd73a26da2c8/volumes"
Nov 03 19:44:55.928859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:55.920478    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2" path="/var/lib/kubelet/pods/f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2/volumes"
Nov 03 19:44:56.588895 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:56.587769    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:57.337296 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:57.336663    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:57.493706 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:57.492582    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:57.818378 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:57.818329    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:58.219914 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:58.219156    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:58.420858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:58.412814    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:58.618863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:58.615420    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:59.443217 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:44:59.443178    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:00.208864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.208795    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz"]
Nov 03 19:45:00.210257 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.210231    5237 cpu_manager.go:414] "containerMap: removing container" podUID="f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2" containerName="maas-api"
Nov 03 19:45:00.210428 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.210414    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2" containerName="maas-api"
Nov 03 19:45:00.210607 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.210593    5237 cpu_manager.go:414] "containerMap: removing container" podUID="0fc24f10-b842-4e07-9da0-d176e93f9f27" containerName="maas-api"
Nov 03 19:45:00.210700 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.210687    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc24f10-b842-4e07-9da0-d176e93f9f27" containerName="maas-api"
Nov 03 19:45:00.210809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.210798    5237 cpu_manager.go:414] "containerMap: removing container" podUID="62e7d6a3-de96-4066-a7dc-bd73a26da2c8" containerName="istio-proxy"
Nov 03 19:45:00.210906 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.210894    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e7d6a3-de96-4066-a7dc-bd73a26da2c8" containerName="istio-proxy"
Nov 03 19:45:00.211162 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.211144    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc24f10-b842-4e07-9da0-d176e93f9f27" containerName="maas-api"
Nov 03 19:45:00.211277 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.211263    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e7d6a3-de96-4066-a7dc-bd73a26da2c8" containerName="istio-proxy"
Nov 03 19:45:00.211379 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.211368    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1447cfd-c1c9-49b7-a9b9-0f3d29d03ea2" containerName="maas-api"
Nov 03 19:45:00.220245 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.220224    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz"
Nov 03 19:45:00.228936 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.228904    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz"]
Nov 03 19:45:00.230555 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.230520    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac7a9c7-b666-4d74-852f-4072b14c6a90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz\" (UID: \"aac7a9c7-b666-4d74-852f-4072b14c6a90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz"
Nov 03 19:45:00.230680 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.230593    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66xv\" (UniqueName: \"kubernetes.io/projected/aac7a9c7-b666-4d74-852f-4072b14c6a90-kube-api-access-f66xv\") pod \"maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz\" (UID: \"aac7a9c7-b666-4d74-852f-4072b14c6a90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz"
Nov 03 19:45:00.290826 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.290733    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\""
Nov 03 19:45:00.332226 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.332192    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac7a9c7-b666-4d74-852f-4072b14c6a90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz\" (UID: \"aac7a9c7-b666-4d74-852f-4072b14c6a90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz"
Nov 03 19:45:00.332417 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.332245    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66xv\" (UniqueName: \"kubernetes.io/projected/aac7a9c7-b666-4d74-852f-4072b14c6a90-kube-api-access-f66xv\") pod \"maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz\" (UID: \"aac7a9c7-b666-4d74-852f-4072b14c6a90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz"
Nov 03 19:45:00.333069 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.333039    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac7a9c7-b666-4d74-852f-4072b14c6a90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz\" (UID: \"aac7a9c7-b666-4d74-852f-4072b14c6a90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz"
Nov 03 19:45:00.345778 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.345750    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66xv\" (UniqueName: \"kubernetes.io/projected/aac7a9c7-b666-4d74-852f-4072b14c6a90-kube-api-access-f66xv\") pod \"maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz\" (UID: \"aac7a9c7-b666-4d74-852f-4072b14c6a90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz"
Nov 03 19:45:00.527702 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.518216    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:00.542465 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.542431    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz"
Nov 03 19:45:00.906786 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.893269    5237 patch_prober.go:29] interesting pod/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:45:00.906786 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:00.893343    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd" podUID="d39b8c6d-00c7-46b5-82a1-a531af3ba320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:01.166325 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:01.166172    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz"]
Nov 03 19:45:01.173948 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:45:01.173915    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac7a9c7_b666_4d74_852f_4072b14c6a90.slice/crio-c185e0cfb3a360ba0ca78b7d7fa4b30aac6625a281f3f314d1431e49ddd0abcc WatchSource:0}: Error finding container c185e0cfb3a360ba0ca78b7d7fa4b30aac6625a281f3f314d1431e49ddd0abcc: Status 404 returned error can't find the container with id c185e0cfb3a360ba0ca78b7d7fa4b30aac6625a281f3f314d1431e49ddd0abcc
Nov 03 19:45:01.373920 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:01.373880    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:01.415354 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:01.415316    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz" event={"ID":"aac7a9c7-b666-4d74-852f-4072b14c6a90","Type":"ContainerStarted","Data":"c185e0cfb3a360ba0ca78b7d7fa4b30aac6625a281f3f314d1431e49ddd0abcc"}
Nov 03 19:45:01.557458 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:01.557415    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:01.716504 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:01.716364    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:01.941670 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:01.941395    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:02.425194 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:02.425161    5237 generic.go:356] "Generic (PLEG): container finished" podID="aac7a9c7-b666-4d74-852f-4072b14c6a90" containerID="302b50f4b0520be990fa7f9d4a5917d714f72459b6f5fd59c577c21963e8fc5c" exitCode=60
Nov 03 19:45:02.425378 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:02.425261    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz" event={"ID":"aac7a9c7-b666-4d74-852f-4072b14c6a90","Type":"ContainerDied","Data":"302b50f4b0520be990fa7f9d4a5917d714f72459b6f5fd59c577c21963e8fc5c"}
Nov 03 19:45:02.425609 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:02.425593    5237 scope.go:122] "RemoveContainer" containerID="302b50f4b0520be990fa7f9d4a5917d714f72459b6f5fd59c577c21963e8fc5c"
Nov 03 19:45:02.714548 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:02.714510    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:02.816875 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:02.816824    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"]
Nov 03 19:45:02.821262 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:02.821236    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"
Nov 03 19:45:02.835776 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:02.833216    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-c54f85\""
Nov 03 19:45:02.836250 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:02.836224    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"]
Nov 03 19:45:03.037256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:02.900308    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7x7p\" (UniqueName: \"kubernetes.io/projected/af9b6d3c-e4c9-478c-9d19-36444a7c4e77-kube-api-access-j7x7p\") pod \"payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr\" (UID: \"af9b6d3c-e4c9-478c-9d19-36444a7c4e77\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"
Nov 03 19:45:03.037256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:02.900551    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9b6d3c-e4c9-478c-9d19-36444a7c4e77-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr\" (UID: \"af9b6d3c-e4c9-478c-9d19-36444a7c4e77\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"
Nov 03 19:45:03.037256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:02.991507    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:03.037256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:03.005660    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7x7p\" (UniqueName: \"kubernetes.io/projected/af9b6d3c-e4c9-478c-9d19-36444a7c4e77-kube-api-access-j7x7p\") pod \"payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr\" (UID: \"af9b6d3c-e4c9-478c-9d19-36444a7c4e77\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"
Nov 03 19:45:03.037256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:03.005858    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9b6d3c-e4c9-478c-9d19-36444a7c4e77-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr\" (UID: \"af9b6d3c-e4c9-478c-9d19-36444a7c4e77\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"
Nov 03 19:45:03.037256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:03.007037    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9b6d3c-e4c9-478c-9d19-36444a7c4e77-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr\" (UID: \"af9b6d3c-e4c9-478c-9d19-36444a7c4e77\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"
Nov 03 19:45:03.156053 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:03.129099    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-c54f85-dockercfg-p28cz\""
Nov 03 19:45:03.241477 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:03.241403    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:03.273540 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:03.269889    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7x7p\" (UniqueName: \"kubernetes.io/projected/af9b6d3c-e4c9-478c-9d19-36444a7c4e77-kube-api-access-j7x7p\") pod \"payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr\" (UID: \"af9b6d3c-e4c9-478c-9d19-36444a7c4e77\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"
Nov 03 19:45:03.273540 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:03.271990    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"]
Nov 03 19:45:03.287914 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:03.287738    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef245bb-72q42"]
Nov 03 19:45:03.438872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:03.438687    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz" event={"ID":"aac7a9c7-b666-4d74-852f-4072b14c6a90","Type":"ContainerStarted","Data":"562e31a1845381ccd53affafb5179adfd8dd0ac99c557e435d09cb89d878ccd0"}
Nov 03 19:45:03.459795 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:03.459410    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz" podStartSLOduration=3.459386925 podStartE2EDuration="3.459386925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:03.457581915 +0000 UTC m=+1059.711889141" watchObservedRunningTime="2026-11-03 19:45:03.459386925 +0000 UTC m=+1059.713694152"
Nov 03 19:45:03.536093 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:03.536045    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"
Nov 03 19:45:03.832673 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:03.832643    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"]
Nov 03 19:45:03.833808 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:45:03.833781    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9b6d3c_e4c9_478c_9d19_36444a7c4e77.slice/crio-1151e55043915b8adaee86195ce323ece9713b05ef687d807ae65346ad5052df WatchSource:0}: Error finding container 1151e55043915b8adaee86195ce323ece9713b05ef687d807ae65346ad5052df: Status 404 returned error can't find the container with id 1151e55043915b8adaee86195ce323ece9713b05ef687d807ae65346ad5052df
Nov 03 19:45:03.861656 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:03.861622    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb" path="/var/lib/kubelet/pods/1cd0cc75-1454-4fe5-a0b9-9fbdddbd04fb/volumes"
Nov 03 19:45:04.038533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:04.038484    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:04.462873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:04.461904    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr" event={"ID":"af9b6d3c-e4c9-478c-9d19-36444a7c4e77","Type":"ContainerStarted","Data":"1151e55043915b8adaee86195ce323ece9713b05ef687d807ae65346ad5052df"}
Nov 03 19:45:04.470296 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:04.464821    5237 generic.go:356] "Generic (PLEG): container finished" podID="aac7a9c7-b666-4d74-852f-4072b14c6a90" containerID="562e31a1845381ccd53affafb5179adfd8dd0ac99c557e435d09cb89d878ccd0" exitCode=60
Nov 03 19:45:04.470296 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:04.464897    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz" event={"ID":"aac7a9c7-b666-4d74-852f-4072b14c6a90","Type":"ContainerDied","Data":"562e31a1845381ccd53affafb5179adfd8dd0ac99c557e435d09cb89d878ccd0"}
Nov 03 19:45:04.470296 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:04.464948    5237 scope.go:122] "RemoveContainer" containerID="302b50f4b0520be990fa7f9d4a5917d714f72459b6f5fd59c577c21963e8fc5c"
Nov 03 19:45:04.470296 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:04.465584    5237 scope.go:122] "RemoveContainer" containerID="562e31a1845381ccd53affafb5179adfd8dd0ac99c557e435d09cb89d878ccd0"
Nov 03 19:45:04.470296 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:45:04.466042    5237 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz_odh-ai-gateway-infra(aac7a9c7-b666-4d74-852f-4072b14c6a90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz" podUID="aac7a9c7-b666-4d74-852f-4072b14c6a90"
Nov 03 19:45:04.878569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:04.878476    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:05.321868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.320920    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"]
Nov 03 19:45:05.321868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.321174    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x" podUID="aa10dff6-a42a-4d78-a4e0-c6828e772701" containerName="maas-api" containerID="cri-o://fc2fb436858a0066314e02e3792d5ad52aba1ae3d765c5e2020a58fb4d453664" gracePeriod=30
Nov 03 19:45:05.407584 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.407546    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:05.491107 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.491067    5237 generic.go:356] "Generic (PLEG): container finished" podID="aa10dff6-a42a-4d78-a4e0-c6828e772701" containerID="fc2fb436858a0066314e02e3792d5ad52aba1ae3d765c5e2020a58fb4d453664" exitCode=0
Nov 03 19:45:05.491288 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.491145    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x" event={"ID":"aa10dff6-a42a-4d78-a4e0-c6828e772701","Type":"ContainerDied","Data":"fc2fb436858a0066314e02e3792d5ad52aba1ae3d765c5e2020a58fb4d453664"}
Nov 03 19:45:05.492727 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.492704    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:45:05.495995 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.494782    5237 scope.go:122] "RemoveContainer" containerID="562e31a1845381ccd53affafb5179adfd8dd0ac99c557e435d09cb89d878ccd0"
Nov 03 19:45:05.495995 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:45:05.495238    5237 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz_odh-ai-gateway-infra(aac7a9c7-b666-4d74-852f-4072b14c6a90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz" podUID="aac7a9c7-b666-4d74-852f-4072b14c6a90"
Nov 03 19:45:05.500203 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.500176    5237 generic.go:356] "Generic (PLEG): container finished" podID="d39b8c6d-00c7-46b5-82a1-a531af3ba320" containerID="a61de96bf7654a462863b676a6af1e392d59ce1e9100ea1e2ac27584f50327e4" exitCode=0
Nov 03 19:45:05.500303 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.500289    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd" event={"ID":"d39b8c6d-00c7-46b5-82a1-a531af3ba320","Type":"ContainerDied","Data":"a61de96bf7654a462863b676a6af1e392d59ce1e9100ea1e2ac27584f50327e4"}
Nov 03 19:45:05.500343 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.500334    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd" event={"ID":"d39b8c6d-00c7-46b5-82a1-a531af3ba320","Type":"ContainerDied","Data":"02275ce096b4fac418d5b52959c29a4afa5bccb5d1e6831ebf18a41ab4e4a3cb"}
Nov 03 19:45:05.500375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.500364    5237 scope.go:122] "RemoveContainer" containerID="a61de96bf7654a462863b676a6af1e392d59ce1e9100ea1e2ac27584f50327e4"
Nov 03 19:45:05.500490 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.500474    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"
Nov 03 19:45:05.514893 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.514867    5237 scope.go:122] "RemoveContainer" containerID="a61de96bf7654a462863b676a6af1e392d59ce1e9100ea1e2ac27584f50327e4"
Nov 03 19:45:05.515284 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:45:05.515257    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61de96bf7654a462863b676a6af1e392d59ce1e9100ea1e2ac27584f50327e4\": container with ID starting with a61de96bf7654a462863b676a6af1e392d59ce1e9100ea1e2ac27584f50327e4 not found: ID does not exist" containerID="a61de96bf7654a462863b676a6af1e392d59ce1e9100ea1e2ac27584f50327e4"
Nov 03 19:45:05.515355 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.515301    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61de96bf7654a462863b676a6af1e392d59ce1e9100ea1e2ac27584f50327e4"} err="failed to get container status \"a61de96bf7654a462863b676a6af1e392d59ce1e9100ea1e2ac27584f50327e4\": rpc error: code = NotFound desc = could not find container \"a61de96bf7654a462863b676a6af1e392d59ce1e9100ea1e2ac27584f50327e4\": container with ID starting with a61de96bf7654a462863b676a6af1e392d59ce1e9100ea1e2ac27584f50327e4 not found: ID does not exist"
Nov 03 19:45:05.597894 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.590382    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-credential-socket\") pod \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") "
Nov 03 19:45:05.597894 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.590537    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istiod-ca-cert\") pod \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") "
Nov 03 19:45:05.597894 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.590648    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-workload-certs\") pod \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") "
Nov 03 19:45:05.597894 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.590678    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-token\" (UniqueName: \"kubernetes.io/projected/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-token\") pod \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") "
Nov 03 19:45:05.597894 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.590719    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-data\") pod \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") "
Nov 03 19:45:05.597894 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.590824    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39b8c6d-00c7-46b5-82a1-a531af3ba320-kube-api-access-crtls\" (UniqueName: \"kubernetes.io/projected/d39b8c6d-00c7-46b5-82a1-a531af3ba320-kube-api-access-crtls\") pod \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") "
Nov 03 19:45:05.597894 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.597226    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-workload-socket\") pod \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") "
Nov 03 19:45:05.597894 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.597426    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-podinfo\") pod \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") "
Nov 03 19:45:05.597894 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.597465    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-envoy\") pod \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\" (UID: \"d39b8c6d-00c7-46b5-82a1-a531af3ba320\") "
Nov 03 19:45:05.618081 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.599031    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-credential-socket" pod "d39b8c6d-00c7-46b5-82a1-a531af3ba320" (UID: "d39b8c6d-00c7-46b5-82a1-a531af3ba320"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:05.618081 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.599376    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istiod-ca-cert" pod "d39b8c6d-00c7-46b5-82a1-a531af3ba320" (UID: "d39b8c6d-00c7-46b5-82a1-a531af3ba320"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:05.618081 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.604544    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-data" pod "d39b8c6d-00c7-46b5-82a1-a531af3ba320" (UID: "d39b8c6d-00c7-46b5-82a1-a531af3ba320"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:05.618081 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.605070    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-workload-socket" pod "d39b8c6d-00c7-46b5-82a1-a531af3ba320" (UID: "d39b8c6d-00c7-46b5-82a1-a531af3ba320"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:05.618081 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.605137    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-workload-certs" pod "d39b8c6d-00c7-46b5-82a1-a531af3ba320" (UID: "d39b8c6d-00c7-46b5-82a1-a531af3ba320"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:05.692119 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.692051    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-envoy" pod "d39b8c6d-00c7-46b5-82a1-a531af3ba320" (UID: "d39b8c6d-00c7-46b5-82a1-a531af3ba320"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:05.694661 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.694505    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:05.700460 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.700328    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:05.700460 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.700362    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:05.700460 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.700378    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:05.700460 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.700392    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:05.700460 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.700406    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:05.700460 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.700419    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:05.800194 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.774711    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39b8c6d-00c7-46b5-82a1-a531af3ba320-kube-api-access-crtls" pod "d39b8c6d-00c7-46b5-82a1-a531af3ba320" (UID: "d39b8c6d-00c7-46b5-82a1-a531af3ba320"). InnerVolumeSpecName "kube-api-access-crtls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:05.800194 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.799032    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-token" pod "d39b8c6d-00c7-46b5-82a1-a531af3ba320" (UID: "d39b8c6d-00c7-46b5-82a1-a531af3ba320"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:05.800194 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.799147    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-podinfo" pod "d39b8c6d-00c7-46b5-82a1-a531af3ba320" (UID: "d39b8c6d-00c7-46b5-82a1-a531af3ba320"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:45:05.924873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.844602    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:05.924873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.889645    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"]
Nov 03 19:45:05.942553 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.941395    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-9b7a388c-openshift-default-7f6db57669-sxrmd"]
Nov 03 19:45:05.942553 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.941478    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:05.942553 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.941496    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crtls\" (UniqueName: \"kubernetes.io/projected/d39b8c6d-00c7-46b5-82a1-a531af3ba320-kube-api-access-crtls\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:05.942553 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:05.941515    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39b8c6d-00c7-46b5-82a1-a531af3ba320-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:06.097159 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.095618    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:45:06.228683 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.228647    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:06.251817 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.251778    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa10dff6-a42a-4d78-a4e0-c6828e772701-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa10dff6-a42a-4d78-a4e0-c6828e772701-openshift-service-ca\") pod \"aa10dff6-a42a-4d78-a4e0-c6828e772701\" (UID: \"aa10dff6-a42a-4d78-a4e0-c6828e772701\") "
Nov 03 19:45:06.252074 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.251898    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa10dff6-a42a-4d78-a4e0-c6828e772701-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa10dff6-a42a-4d78-a4e0-c6828e772701-maas-api-tls\") pod \"aa10dff6-a42a-4d78-a4e0-c6828e772701\" (UID: \"aa10dff6-a42a-4d78-a4e0-c6828e772701\") "
Nov 03 19:45:06.252155 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.252104    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa10dff6-a42a-4d78-a4e0-c6828e772701-kube-api-access-97v5l\" (UniqueName: \"kubernetes.io/projected/aa10dff6-a42a-4d78-a4e0-c6828e772701-kube-api-access-97v5l\") pod \"aa10dff6-a42a-4d78-a4e0-c6828e772701\" (UID: \"aa10dff6-a42a-4d78-a4e0-c6828e772701\") "
Nov 03 19:45:06.252227 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.252182    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa10dff6-a42a-4d78-a4e0-c6828e772701-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa10dff6-a42a-4d78-a4e0-c6828e772701-metrics-certs\") pod \"aa10dff6-a42a-4d78-a4e0-c6828e772701\" (UID: \"aa10dff6-a42a-4d78-a4e0-c6828e772701\") "
Nov 03 19:45:06.272367 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.253959    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa10dff6-a42a-4d78-a4e0-c6828e772701-openshift-service-ca" pod "aa10dff6-a42a-4d78-a4e0-c6828e772701" (UID: "aa10dff6-a42a-4d78-a4e0-c6828e772701"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:06.366777 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.359165    5237 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa10dff6-a42a-4d78-a4e0-c6828e772701-openshift-service-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:06.366777 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.359684    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa10dff6-a42a-4d78-a4e0-c6828e772701-maas-api-tls" pod "aa10dff6-a42a-4d78-a4e0-c6828e772701" (UID: "aa10dff6-a42a-4d78-a4e0-c6828e772701"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:45:06.366777 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.359782    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa10dff6-a42a-4d78-a4e0-c6828e772701-kube-api-access-97v5l" pod "aa10dff6-a42a-4d78-a4e0-c6828e772701" (UID: "aa10dff6-a42a-4d78-a4e0-c6828e772701"). InnerVolumeSpecName "kube-api-access-97v5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:06.397928 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.393997    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa10dff6-a42a-4d78-a4e0-c6828e772701-metrics-certs" pod "aa10dff6-a42a-4d78-a4e0-c6828e772701" (UID: "aa10dff6-a42a-4d78-a4e0-c6828e772701"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:45:06.460888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.460804    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa10dff6-a42a-4d78-a4e0-c6828e772701-maas-api-tls\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:06.461133 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.461113    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97v5l\" (UniqueName: \"kubernetes.io/projected/aa10dff6-a42a-4d78-a4e0-c6828e772701-kube-api-access-97v5l\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:06.461237 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.461217    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa10dff6-a42a-4d78-a4e0-c6828e772701-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:06.511334 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.511296    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x" event={"ID":"aa10dff6-a42a-4d78-a4e0-c6828e772701","Type":"ContainerDied","Data":"147966900cc10d5f6443a500ba100859f3502369f5b0cca112f6b397e22f2224"}
Nov 03 19:45:06.511685 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.511660    5237 scope.go:122] "RemoveContainer" containerID="fc2fb436858a0066314e02e3792d5ad52aba1ae3d765c5e2020a58fb4d453664"
Nov 03 19:45:06.511807 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.511353    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"
Nov 03 19:45:06.544285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.544236    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"]
Nov 03 19:45:06.549262 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:06.549224    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14a79569-7c8f5fcd59-qsw6x"]
Nov 03 19:45:07.025294 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:07.025255    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:07.734914 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:07.734873    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"]
Nov 03 19:45:07.735189 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:07.735154    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp" podUID="42eef399-4ea6-4e43-b54f-b00c9cce32f7" containerName="istio-proxy" containerID="cri-o://0b5de6f9d3b4c5d73301acd6a584fe4fc7bff2d763f3cd4ef3a603280eab5c55" gracePeriod=30
Nov 03 19:45:07.796036 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:07.795993    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:07.993567 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:07.993480    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa10dff6-a42a-4d78-a4e0-c6828e772701" path="/var/lib/kubelet/pods/aa10dff6-a42a-4d78-a4e0-c6828e772701/volumes"
Nov 03 19:45:08.001023 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.000984    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d39b8c6d-00c7-46b5-82a1-a531af3ba320" path="/var/lib/kubelet/pods/d39b8c6d-00c7-46b5-82a1-a531af3ba320/volumes"
Nov 03 19:45:08.038888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.030234    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:08.531623 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.531581    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr" event={"ID":"af9b6d3c-e4c9-478c-9d19-36444a7c4e77","Type":"ContainerStarted","Data":"aef580658217fa9e7afb2f3b3555c062a4d1e1efa85b68e99b8abde0bb95d4c1"}
Nov 03 19:45:08.531805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.531680    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"
Nov 03 19:45:08.555763 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.555709    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr" podStartSLOduration=2.745022505 podStartE2EDuration="6.555694654s" totalImagesPullingTime="3.810672149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:08.552219634 +0000 UTC m=+1064.806526858" watchObservedRunningTime="2026-11-03 19:45:08.555694654 +0000 UTC m=+1064.810001879"
Nov 03 19:45:08.613995 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.613960    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"]
Nov 03 19:45:08.614645 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.614624    5237 cpu_manager.go:414] "containerMap: removing container" podUID="d39b8c6d-00c7-46b5-82a1-a531af3ba320" containerName="istio-proxy"
Nov 03 19:45:08.614777 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.614646    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d39b8c6d-00c7-46b5-82a1-a531af3ba320" containerName="istio-proxy"
Nov 03 19:45:08.614777 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.614683    5237 cpu_manager.go:414] "containerMap: removing container" podUID="aa10dff6-a42a-4d78-a4e0-c6828e772701" containerName="maas-api"
Nov 03 19:45:08.614777 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.614691    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa10dff6-a42a-4d78-a4e0-c6828e772701" containerName="maas-api"
Nov 03 19:45:08.614976 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.614831    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="d39b8c6d-00c7-46b5-82a1-a531af3ba320" containerName="istio-proxy"
Nov 03 19:45:08.614976 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.614864    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa10dff6-a42a-4d78-a4e0-c6828e772701" containerName="maas-api"
Nov 03 19:45:08.619637 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.619617    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:45:08.638115 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.638076    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-c54f85-serving-cert\""
Nov 03 19:45:08.642493 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.642464    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"]
Nov 03 19:45:08.671719 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.671681    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:08.730748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.730437    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdwv2\" (UniqueName: \"kubernetes.io/projected/9145fea2-1fed-40d6-82e4-2f79350f9d8d-kube-api-access-zdwv2\") pod \"maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz\" (UID: \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:45:08.730748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.730503    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9145fea2-1fed-40d6-82e4-2f79350f9d8d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz\" (UID: \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:45:08.730748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.730546    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9145fea2-1fed-40d6-82e4-2f79350f9d8d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz\" (UID: \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:45:08.730748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.730580    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9145fea2-1fed-40d6-82e4-2f79350f9d8d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz\" (UID: \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:45:08.819858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.818821    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:08.834869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.834168    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9145fea2-1fed-40d6-82e4-2f79350f9d8d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz\" (UID: \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:45:08.834869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.834227    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9145fea2-1fed-40d6-82e4-2f79350f9d8d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz\" (UID: \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:45:08.834869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.834269    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9145fea2-1fed-40d6-82e4-2f79350f9d8d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz\" (UID: \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:45:08.834869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.834810    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdwv2\" (UniqueName: \"kubernetes.io/projected/9145fea2-1fed-40d6-82e4-2f79350f9d8d-kube-api-access-zdwv2\") pod \"maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz\" (UID: \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:45:08.853920 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.838810    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9145fea2-1fed-40d6-82e4-2f79350f9d8d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz\" (UID: \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:45:08.861319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.859295    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9145fea2-1fed-40d6-82e4-2f79350f9d8d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz\" (UID: \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:45:08.889944 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.886609    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdwv2\" (UniqueName: \"kubernetes.io/projected/9145fea2-1fed-40d6-82e4-2f79350f9d8d-kube-api-access-zdwv2\") pod \"maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz\" (UID: \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:45:08.909601 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.909569    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9145fea2-1fed-40d6-82e4-2f79350f9d8d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz\" (UID: \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:45:08.938890 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:08.938823    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:45:09.039225 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.039047    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"]
Nov 03 19:45:09.057928 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.057752    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.064029 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.057642    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"]
Nov 03 19:45:09.074580 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.072097    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-763a69-openshift-default-dockercfg-wgvkh\""
Nov 03 19:45:09.131530 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.131492    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:09.263474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.262120    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-data\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.263474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.262176    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-envoy\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.263474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.262216    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-token\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.263474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.262255    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j49mz\" (UniqueName: \"kubernetes.io/projected/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-kube-api-access-j49mz\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.263474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.262382    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-workload-socket\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.263474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.262491    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-podinfo\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.263474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.262524    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-credential-socket\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.263474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.262629    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-workload-certs\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.263474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.262705    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istiod-ca-cert\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.356753 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.356668    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:09.411538 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.371187    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-workload-socket\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.411538 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.371292    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-podinfo\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.411538 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.371324    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-credential-socket\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.411538 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.371407    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-workload-certs\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.411538 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.371489    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istiod-ca-cert\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.411538 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.385751    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-credential-socket\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.411538 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.409024    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-data\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.411538 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.409074    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-envoy\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.411538 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.409114    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-token\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.411538 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.409156    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j49mz\" (UniqueName: \"kubernetes.io/projected/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-kube-api-access-j49mz\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.411538 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.410307    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-workload-socket\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.411538 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.410545    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-workload-certs\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.411538 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.411179    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istiod-ca-cert\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.412860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.412794    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-data\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.463857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.426011    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"]
Nov 03 19:45:09.463857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.433486    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-podinfo\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.463857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.437114    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-envoy\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.487860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.476888    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j49mz\" (UniqueName: \"kubernetes.io/projected/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-kube-api-access-j49mz\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.505684 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.505611    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-token\") pod \"e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") " pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.692168 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.692056    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:09.726898 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.725504    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz" event={"ID":"9145fea2-1fed-40d6-82e4-2f79350f9d8d","Type":"ContainerStarted","Data":"3141ef8f945c4d8d5494e85348516a76e3ba63634788e5cf35450aeea632d591"}
Nov 03 19:45:09.989088 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.988634    5237 patch_prober.go:29] interesting pod/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:45:09.989088 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:09.988735    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp" podUID="42eef399-4ea6-4e43-b54f-b00c9cce32f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:45:10.350497 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:10.350465    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:10.482375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:10.482336    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"]
Nov 03 19:45:10.506085 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:45:10.506025    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68bcd4b4_76d8_4e68_ac9c_cabeaa560b7a.slice/crio-66e7c91815ce8fd34b9c97c6ab84917c2f4725cb4425632ed2772d12daa5051f WatchSource:0}: Error finding container 66e7c91815ce8fd34b9c97c6ab84917c2f4725cb4425632ed2772d12daa5051f: Status 404 returned error can't find the container with id 66e7c91815ce8fd34b9c97c6ab84917c2f4725cb4425632ed2772d12daa5051f
Nov 03 19:45:10.533236 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:10.530953    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:45:10.533236 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:10.531054    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:45:10.533236 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:10.531155    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:45:10.733948 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:10.733907    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6" event={"ID":"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a","Type":"ContainerStarted","Data":"7ec9b427e11b45b9ede06afc80f5789fcf3b67c99b2780c46cb5e4d248623663"}
Nov 03 19:45:10.734124 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:10.733979    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6" event={"ID":"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a","Type":"ContainerStarted","Data":"66e7c91815ce8fd34b9c97c6ab84917c2f4725cb4425632ed2772d12daa5051f"}
Nov 03 19:45:10.736201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:10.736173    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz" event={"ID":"9145fea2-1fed-40d6-82e4-2f79350f9d8d","Type":"ContainerStarted","Data":"34ec9b321753646cd6a71ed518e1662a6544b7031fc7eabf7ff2966676b9ca41"}
Nov 03 19:45:10.736364 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:10.736242    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:45:10.762402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:10.762251    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6" podStartSLOduration=1.762227299 podStartE2EDuration="1.762227299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:10.759124812 +0000 UTC m=+1067.013432037" watchObservedRunningTime="2026-11-03 19:45:10.762227299 +0000 UTC m=+1067.016534526"
Nov 03 19:45:10.782195 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:10.782137    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz" podStartSLOduration=2.059974818 podStartE2EDuration="2.782121162s" totalImagesPullingTime="722.146344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:10.780359895 +0000 UTC m=+1067.034667120" watchObservedRunningTime="2026-11-03 19:45:10.782121162 +0000 UTC m=+1067.036428389"
Nov 03 19:45:11.338860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:11.335294    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:11.693591 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:11.693488    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:11.699481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:11.699451    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:11.742641 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:11.742606    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:11.743472 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:11.743450    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:45:11.935154 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:11.935122    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:12.724150 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:12.724111    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:13.208012 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.207971    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:13.257435 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:45:13.257402    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:45:13.380973 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.376447    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:13.470225 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.470151    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:45:13.551678 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.551647    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42eef399-4ea6-4e43-b54f-b00c9cce32f7-kube-api-access-ks788\" (UniqueName: \"kubernetes.io/projected/42eef399-4ea6-4e43-b54f-b00c9cce32f7-kube-api-access-ks788\") pod \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") "
Nov 03 19:45:13.551858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.551802    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-workload-socket\") pod \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") "
Nov 03 19:45:13.551858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.551853    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istiod-ca-cert\") pod \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") "
Nov 03 19:45:13.552167 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.552146    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-envoy\") pod \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") "
Nov 03 19:45:13.552167 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.552183    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-credential-socket\") pod \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") "
Nov 03 19:45:13.552167 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.552213    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-data\") pod \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") "
Nov 03 19:45:13.552439 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.552267    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-workload-certs\") pod \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") "
Nov 03 19:45:13.552439 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.552359    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istiod-ca-cert" pod "42eef399-4ea6-4e43-b54f-b00c9cce32f7" (UID: "42eef399-4ea6-4e43-b54f-b00c9cce32f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:13.552439 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.552381    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-podinfo\") pod \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") "
Nov 03 19:45:13.552439 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.552431    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-token\" (UniqueName: \"kubernetes.io/projected/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-token\") pod \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\" (UID: \"42eef399-4ea6-4e43-b54f-b00c9cce32f7\") "
Nov 03 19:45:13.553306 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.552642    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-data" pod "42eef399-4ea6-4e43-b54f-b00c9cce32f7" (UID: "42eef399-4ea6-4e43-b54f-b00c9cce32f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:13.553306 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.553259    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-workload-certs" pod "42eef399-4ea6-4e43-b54f-b00c9cce32f7" (UID: "42eef399-4ea6-4e43-b54f-b00c9cce32f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:13.553568 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.553511    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:13.553568 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.553557    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:13.553796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.553702    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-workload-socket" pod "42eef399-4ea6-4e43-b54f-b00c9cce32f7" (UID: "42eef399-4ea6-4e43-b54f-b00c9cce32f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:13.553980 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.553940    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-credential-socket" pod "42eef399-4ea6-4e43-b54f-b00c9cce32f7" (UID: "42eef399-4ea6-4e43-b54f-b00c9cce32f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:13.555668 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.555640    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-envoy" pod "42eef399-4ea6-4e43-b54f-b00c9cce32f7" (UID: "42eef399-4ea6-4e43-b54f-b00c9cce32f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:13.555779 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.555757    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-token" pod "42eef399-4ea6-4e43-b54f-b00c9cce32f7" (UID: "42eef399-4ea6-4e43-b54f-b00c9cce32f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:13.556191 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.556162    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-podinfo" pod "42eef399-4ea6-4e43-b54f-b00c9cce32f7" (UID: "42eef399-4ea6-4e43-b54f-b00c9cce32f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:45:13.562235 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.562213    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42eef399-4ea6-4e43-b54f-b00c9cce32f7-kube-api-access-ks788" pod "42eef399-4ea6-4e43-b54f-b00c9cce32f7" (UID: "42eef399-4ea6-4e43-b54f-b00c9cce32f7"). InnerVolumeSpecName "kube-api-access-ks788". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:13.670930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.664909    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:13.670930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.669250    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:13.670930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.669283    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:13.670930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.669298    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:13.670930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.669312    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42eef399-4ea6-4e43-b54f-b00c9cce32f7-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:13.670930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.669326    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:13.670930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.669341    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42eef399-4ea6-4e43-b54f-b00c9cce32f7-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:13.670930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.669354    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks788\" (UniqueName: \"kubernetes.io/projected/42eef399-4ea6-4e43-b54f-b00c9cce32f7-kube-api-access-ks788\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:13.885197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.885167    5237 generic.go:356] "Generic (PLEG): container finished" podID="42eef399-4ea6-4e43-b54f-b00c9cce32f7" containerID="0b5de6f9d3b4c5d73301acd6a584fe4fc7bff2d763f3cd4ef3a603280eab5c55" exitCode=0
Nov 03 19:45:13.885582 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.885210    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"
Nov 03 19:45:13.885582 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.885252    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp" event={"ID":"42eef399-4ea6-4e43-b54f-b00c9cce32f7","Type":"ContainerDied","Data":"0b5de6f9d3b4c5d73301acd6a584fe4fc7bff2d763f3cd4ef3a603280eab5c55"}
Nov 03 19:45:13.885582 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.885307    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp" event={"ID":"42eef399-4ea6-4e43-b54f-b00c9cce32f7","Type":"ContainerDied","Data":"b207a6eb60b996080fb3509cb069e1898d09886052d335abd4b96b5ca5fc5ff3"}
Nov 03 19:45:13.885582 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.885339    5237 scope.go:122] "RemoveContainer" containerID="0b5de6f9d3b4c5d73301acd6a584fe4fc7bff2d763f3cd4ef3a603280eab5c55"
Nov 03 19:45:13.896136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.896103    5237 scope.go:122] "RemoveContainer" containerID="0b5de6f9d3b4c5d73301acd6a584fe4fc7bff2d763f3cd4ef3a603280eab5c55"
Nov 03 19:45:13.896440 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:45:13.896418    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b5de6f9d3b4c5d73301acd6a584fe4fc7bff2d763f3cd4ef3a603280eab5c55\": container with ID starting with 0b5de6f9d3b4c5d73301acd6a584fe4fc7bff2d763f3cd4ef3a603280eab5c55 not found: ID does not exist" containerID="0b5de6f9d3b4c5d73301acd6a584fe4fc7bff2d763f3cd4ef3a603280eab5c55"
Nov 03 19:45:13.896485 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.896449    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b5de6f9d3b4c5d73301acd6a584fe4fc7bff2d763f3cd4ef3a603280eab5c55"} err="failed to get container status \"0b5de6f9d3b4c5d73301acd6a584fe4fc7bff2d763f3cd4ef3a603280eab5c55\": rpc error: code = NotFound desc = could not find container \"0b5de6f9d3b4c5d73301acd6a584fe4fc7bff2d763f3cd4ef3a603280eab5c55\": container with ID starting with 0b5de6f9d3b4c5d73301acd6a584fe4fc7bff2d763f3cd4ef3a603280eab5c55 not found: ID does not exist"
Nov 03 19:45:13.905727 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.905698    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"]
Nov 03 19:45:13.910258 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:13.910234    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-14a79569-openshift-default-56d8ff8b9c-hnkmp"]
Nov 03 19:45:14.477395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.477362    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:14.550784 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.550754    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"]
Nov 03 19:45:14.551465 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.551448    5237 cpu_manager.go:414] "containerMap: removing container" podUID="42eef399-4ea6-4e43-b54f-b00c9cce32f7" containerName="istio-proxy"
Nov 03 19:45:14.551465 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.551464    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42eef399-4ea6-4e43-b54f-b00c9cce32f7" containerName="istio-proxy"
Nov 03 19:45:14.551595 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.551571    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="42eef399-4ea6-4e43-b54f-b00c9cce32f7" containerName="istio-proxy"
Nov 03 19:45:14.556416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.556394    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"
Nov 03 19:45:14.569191 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.569144    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-763a69-dockercfg-62pzz\""
Nov 03 19:45:14.569339 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.569283    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-763a69\""
Nov 03 19:45:14.573662 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.572858    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"]
Nov 03 19:45:14.655909 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.655873    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:14.698859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.696987    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3882ca5a-6385-41e0-892c-21f22d58ddc9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227\" (UID: \"3882ca5a-6385-41e0-892c-21f22d58ddc9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"
Nov 03 19:45:14.698859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.697210    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gbb4\" (UniqueName: \"kubernetes.io/projected/3882ca5a-6385-41e0-892c-21f22d58ddc9-kube-api-access-2gbb4\") pod \"payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227\" (UID: \"3882ca5a-6385-41e0-892c-21f22d58ddc9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"
Nov 03 19:45:14.802857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.799876    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3882ca5a-6385-41e0-892c-21f22d58ddc9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227\" (UID: \"3882ca5a-6385-41e0-892c-21f22d58ddc9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"
Nov 03 19:45:14.802857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.800162    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gbb4\" (UniqueName: \"kubernetes.io/projected/3882ca5a-6385-41e0-892c-21f22d58ddc9-kube-api-access-2gbb4\") pod \"payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227\" (UID: \"3882ca5a-6385-41e0-892c-21f22d58ddc9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"
Nov 03 19:45:14.802857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.801582    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3882ca5a-6385-41e0-892c-21f22d58ddc9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227\" (UID: \"3882ca5a-6385-41e0-892c-21f22d58ddc9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"
Nov 03 19:45:14.826855 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.825811    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gbb4\" (UniqueName: \"kubernetes.io/projected/3882ca5a-6385-41e0-892c-21f22d58ddc9-kube-api-access-2gbb4\") pod \"payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227\" (UID: \"3882ca5a-6385-41e0-892c-21f22d58ddc9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"
Nov 03 19:45:14.861861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.854566    5237 scope.go:122] "RemoveContainer" containerID="562e31a1845381ccd53affafb5179adfd8dd0ac99c557e435d09cb89d878ccd0"
Nov 03 19:45:14.885046 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:14.884080    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"
Nov 03 19:45:15.346903 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:15.346874    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"]
Nov 03 19:45:15.348792 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:45:15.348763    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3882ca5a_6385_41e0_892c_21f22d58ddc9.slice/crio-932ff957e35f5cb79c13414482493bc41ee0e60f487dd105d5be2f42bb8809b7 WatchSource:0}: Error finding container 932ff957e35f5cb79c13414482493bc41ee0e60f487dd105d5be2f42bb8809b7: Status 404 returned error can't find the container with id 932ff957e35f5cb79c13414482493bc41ee0e60f487dd105d5be2f42bb8809b7
Nov 03 19:45:15.424177 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:15.424147    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:15.870531 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:15.870496    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42eef399-4ea6-4e43-b54f-b00c9cce32f7" path="/var/lib/kubelet/pods/42eef399-4ea6-4e43-b54f-b00c9cce32f7/volumes"
Nov 03 19:45:15.983940 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:15.983910    5237 generic.go:356] "Generic (PLEG): container finished" podID="aac7a9c7-b666-4d74-852f-4072b14c6a90" containerID="4ef61d544b495fa05521d6181a1b6c5cc2080a3ac25a0ad71056b42ff74a5ac5" exitCode=60
Nov 03 19:45:15.984049 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:15.983977    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz" event={"ID":"aac7a9c7-b666-4d74-852f-4072b14c6a90","Type":"ContainerDied","Data":"4ef61d544b495fa05521d6181a1b6c5cc2080a3ac25a0ad71056b42ff74a5ac5"}
Nov 03 19:45:15.984117 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:15.984052    5237 scope.go:122] "RemoveContainer" containerID="562e31a1845381ccd53affafb5179adfd8dd0ac99c557e435d09cb89d878ccd0"
Nov 03 19:45:15.984500 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:15.984479    5237 scope.go:122] "RemoveContainer" containerID="4ef61d544b495fa05521d6181a1b6c5cc2080a3ac25a0ad71056b42ff74a5ac5"
Nov 03 19:45:15.984995 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:45:15.984966    5237 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz_odh-ai-gateway-infra(aac7a9c7-b666-4d74-852f-4072b14c6a90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz" podUID="aac7a9c7-b666-4d74-852f-4072b14c6a90"
Nov 03 19:45:15.986010 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:15.985989    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227" event={"ID":"3882ca5a-6385-41e0-892c-21f22d58ddc9","Type":"ContainerStarted","Data":"cfd97b8b32ad10cb8935759925e24ac4782719a685c08cfd998e432968cbb223"}
Nov 03 19:45:15.986089 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:15.986053    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227" event={"ID":"3882ca5a-6385-41e0-892c-21f22d58ddc9","Type":"ContainerStarted","Data":"932ff957e35f5cb79c13414482493bc41ee0e60f487dd105d5be2f42bb8809b7"}
Nov 03 19:45:15.986644 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:15.986627    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"
Nov 03 19:45:16.035790 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:16.035732    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227" podStartSLOduration=2.035717352 podStartE2EDuration="2.035717352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:16.032064379 +0000 UTC m=+1072.286371594" watchObservedRunningTime="2026-11-03 19:45:16.035717352 +0000 UTC m=+1072.290024578"
Nov 03 19:45:16.321726 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:16.321227    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:16.799563 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:16.799150    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:45:17.024815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:17.024783    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz"]
Nov 03 19:45:17.070591 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:17.070487    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:17.338535 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:17.338459    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz"
Nov 03 19:45:17.437951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:17.437918    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac7a9c7-b666-4d74-852f-4072b14c6a90-kube-api-access-f66xv\" (UniqueName: \"kubernetes.io/projected/aac7a9c7-b666-4d74-852f-4072b14c6a90-kube-api-access-f66xv\") pod \"aac7a9c7-b666-4d74-852f-4072b14c6a90\" (UID: \"aac7a9c7-b666-4d74-852f-4072b14c6a90\") "
Nov 03 19:45:17.438176 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:17.438084    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aac7a9c7-b666-4d74-852f-4072b14c6a90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac7a9c7-b666-4d74-852f-4072b14c6a90-maas-api-service-ca\") pod \"aac7a9c7-b666-4d74-852f-4072b14c6a90\" (UID: \"aac7a9c7-b666-4d74-852f-4072b14c6a90\") "
Nov 03 19:45:17.438454 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:17.438430    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac7a9c7-b666-4d74-852f-4072b14c6a90-maas-api-service-ca" pod "aac7a9c7-b666-4d74-852f-4072b14c6a90" (UID: "aac7a9c7-b666-4d74-852f-4072b14c6a90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:17.438759 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:17.438741    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac7a9c7-b666-4d74-852f-4072b14c6a90-maas-api-service-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:17.440286 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:17.440260    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac7a9c7-b666-4d74-852f-4072b14c6a90-kube-api-access-f66xv" pod "aac7a9c7-b666-4d74-852f-4072b14c6a90" (UID: "aac7a9c7-b666-4d74-852f-4072b14c6a90"). InnerVolumeSpecName "kube-api-access-f66xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:17.539937 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:17.539898    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f66xv\" (UniqueName: \"kubernetes.io/projected/aac7a9c7-b666-4d74-852f-4072b14c6a90-kube-api-access-f66xv\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:18.004824 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:18.004796    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz"
Nov 03 19:45:18.005300 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:18.004816    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz" event={"ID":"aac7a9c7-b666-4d74-852f-4072b14c6a90","Type":"ContainerDied","Data":"c185e0cfb3a360ba0ca78b7d7fa4b30aac6625a281f3f314d1431e49ddd0abcc"}
Nov 03 19:45:18.005300 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:18.004910    5237 scope.go:122] "RemoveContainer" containerID="4ef61d544b495fa05521d6181a1b6c5cc2080a3ac25a0ad71056b42ff74a5ac5"
Nov 03 19:45:18.053860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:18.043673    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:18.053860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:18.044911    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz"]
Nov 03 19:45:18.053860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:18.050284    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ef8f84-29895585-84grz"]
Nov 03 19:45:18.837889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:18.837572    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:19.354670 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:19.354637    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:19.493853 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:19.493788    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:19.654407 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:19.654322    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:19.728583 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:19.728555    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"
Nov 03 19:45:19.865517 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:19.863167    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aac7a9c7-b666-4d74-852f-4072b14c6a90" path="/var/lib/kubelet/pods/aac7a9c7-b666-4d74-852f-4072b14c6a90/volumes"
Nov 03 19:45:20.243546 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.243517    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"]
Nov 03 19:45:20.244439 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.244420    5237 cpu_manager.go:414] "containerMap: removing container" podUID="aac7a9c7-b666-4d74-852f-4072b14c6a90" containerName="cleanup"
Nov 03 19:45:20.244541 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.244441    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac7a9c7-b666-4d74-852f-4072b14c6a90" containerName="cleanup"
Nov 03 19:45:20.244541 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.244458    5237 cpu_manager.go:414] "containerMap: removing container" podUID="aac7a9c7-b666-4d74-852f-4072b14c6a90" containerName="cleanup"
Nov 03 19:45:20.244541 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.244464    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac7a9c7-b666-4d74-852f-4072b14c6a90" containerName="cleanup"
Nov 03 19:45:20.244541 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.244483    5237 cpu_manager.go:414] "containerMap: removing container" podUID="aac7a9c7-b666-4d74-852f-4072b14c6a90" containerName="cleanup"
Nov 03 19:45:20.244541 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.244488    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac7a9c7-b666-4d74-852f-4072b14c6a90" containerName="cleanup"
Nov 03 19:45:20.244850 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.244575    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac7a9c7-b666-4d74-852f-4072b14c6a90" containerName="cleanup"
Nov 03 19:45:20.244850 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.244585    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac7a9c7-b666-4d74-852f-4072b14c6a90" containerName="cleanup"
Nov 03 19:45:20.253352 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.251912    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:45:20.265914 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.265890    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-763a69-serving-cert\""
Nov 03 19:45:20.271587 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.271560    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"]
Nov 03 19:45:20.313905 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.313831    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bee0fc-7374-4419-ac8a-07745bdc3a24-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7\" (UID: \"93bee0fc-7374-4419-ac8a-07745bdc3a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:45:20.314120 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.314093    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93bee0fc-7374-4419-ac8a-07745bdc3a24-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7\" (UID: \"93bee0fc-7374-4419-ac8a-07745bdc3a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:45:20.314211 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.314190    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45zgg\" (UniqueName: \"kubernetes.io/projected/93bee0fc-7374-4419-ac8a-07745bdc3a24-kube-api-access-45zgg\") pod \"maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7\" (UID: \"93bee0fc-7374-4419-ac8a-07745bdc3a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:45:20.314327 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.314308    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93bee0fc-7374-4419-ac8a-07745bdc3a24-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7\" (UID: \"93bee0fc-7374-4419-ac8a-07745bdc3a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:45:20.396863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.396180    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:20.421906 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.419735    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93bee0fc-7374-4419-ac8a-07745bdc3a24-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7\" (UID: \"93bee0fc-7374-4419-ac8a-07745bdc3a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:45:20.437857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.432502    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bee0fc-7374-4419-ac8a-07745bdc3a24-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7\" (UID: \"93bee0fc-7374-4419-ac8a-07745bdc3a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:45:20.437857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.432709    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93bee0fc-7374-4419-ac8a-07745bdc3a24-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7\" (UID: \"93bee0fc-7374-4419-ac8a-07745bdc3a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:45:20.437857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.432793    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45zgg\" (UniqueName: \"kubernetes.io/projected/93bee0fc-7374-4419-ac8a-07745bdc3a24-kube-api-access-45zgg\") pod \"maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7\" (UID: \"93bee0fc-7374-4419-ac8a-07745bdc3a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:45:20.437857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.433348    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bee0fc-7374-4419-ac8a-07745bdc3a24-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7\" (UID: \"93bee0fc-7374-4419-ac8a-07745bdc3a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:45:20.440419 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.439447    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93bee0fc-7374-4419-ac8a-07745bdc3a24-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7\" (UID: \"93bee0fc-7374-4419-ac8a-07745bdc3a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:45:20.477913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.477135    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93bee0fc-7374-4419-ac8a-07745bdc3a24-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7\" (UID: \"93bee0fc-7374-4419-ac8a-07745bdc3a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:45:20.495906 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.491787    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45zgg\" (UniqueName: \"kubernetes.io/projected/93bee0fc-7374-4419-ac8a-07745bdc3a24-kube-api-access-45zgg\") pod \"maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7\" (UID: \"93bee0fc-7374-4419-ac8a-07745bdc3a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:45:20.542604 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.542567    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:20.568993 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.568958    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:45:20.713483 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.713438    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:20.878109 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:20.878074    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"]
Nov 03 19:45:20.895828 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:45:20.895796    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bee0fc_7374_4419_ac8a_07745bdc3a24.slice/crio-59fe8b164f1f78fc68ba3a719badf5f2696c50b94a2b55730cf4fa7f568a2f96 WatchSource:0}: Error finding container 59fe8b164f1f78fc68ba3a719badf5f2696c50b94a2b55730cf4fa7f568a2f96: Status 404 returned error can't find the container with id 59fe8b164f1f78fc68ba3a719badf5f2696c50b94a2b55730cf4fa7f568a2f96
Nov 03 19:45:21.090862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:21.087270    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7" event={"ID":"93bee0fc-7374-4419-ac8a-07745bdc3a24","Type":"ContainerStarted","Data":"59fe8b164f1f78fc68ba3a719badf5f2696c50b94a2b55730cf4fa7f568a2f96"}
Nov 03 19:45:21.526010 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:21.525977    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:21.682930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:21.682764    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:22.096150 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:22.096117    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7" event={"ID":"93bee0fc-7374-4419-ac8a-07745bdc3a24","Type":"ContainerStarted","Data":"2049428a0e150b3e75e258dd7bf2abf7fabd6795db7445d016203ee05185aff3"}
Nov 03 19:45:22.096350 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:22.096203    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:45:22.101423 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:22.101396    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:22.118884 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:22.118820    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7" podStartSLOduration=1.6426677029999999 podStartE2EDuration="2.118805609s" totalImagesPullingTime="476.137906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:22.117150805 +0000 UTC m=+1078.371458029" watchObservedRunningTime="2026-11-03 19:45:22.118805609 +0000 UTC m=+1078.373112891"
Nov 03 19:45:22.407297 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:22.407064    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:22.640924 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:22.628078    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:23.014872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:23.005286    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:23.639857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:23.635578    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:24.685858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:24.684088    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:25.557734 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:25.557687    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:26.220465 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:26.220430    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:26.445494 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:26.445457    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:26.578502 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:26.578463    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:27.000036 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:26.999951    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"
Nov 03 19:45:27.198719 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.198680    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:27.671935 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.671896    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:27.721034 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.720997    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"]
Nov 03 19:45:27.721758 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.721741    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac7a9c7-b666-4d74-852f-4072b14c6a90" containerName="cleanup"
Nov 03 19:45:27.726871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.726822    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"
Nov 03 19:45:27.735406 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.735379    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-2b2e60-dockercfg-44qdk\""
Nov 03 19:45:27.740258 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.740230    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"]
Nov 03 19:45:27.761911 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.761880    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"]
Nov 03 19:45:27.766104 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.766086    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"
Nov 03 19:45:27.769878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.769825    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwqkw\" (UniqueName: \"kubernetes.io/projected/fe7148f3-a2df-43bb-bf04-d5a0fddc1a13-kube-api-access-fwqkw\") pod \"payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd\" (UID: \"fe7148f3-a2df-43bb-bf04-d5a0fddc1a13\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"
Nov 03 19:45:27.770009 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.769963    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7148f3-a2df-43bb-bf04-d5a0fddc1a13-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd\" (UID: \"fe7148f3-a2df-43bb-bf04-d5a0fddc1a13\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"
Nov 03 19:45:27.776824 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.776799    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"]
Nov 03 19:45:27.786142 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.786116    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-2b2e60\""
Nov 03 19:45:27.896292 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.896256    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:27.951916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.928850    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwqkw\" (UniqueName: \"kubernetes.io/projected/fe7148f3-a2df-43bb-bf04-d5a0fddc1a13-kube-api-access-fwqkw\") pod \"payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd\" (UID: \"fe7148f3-a2df-43bb-bf04-d5a0fddc1a13\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"
Nov 03 19:45:27.951916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.928970    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7148f3-a2df-43bb-bf04-d5a0fddc1a13-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd\" (UID: \"fe7148f3-a2df-43bb-bf04-d5a0fddc1a13\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"
Nov 03 19:45:27.951916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.929166    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4prd7\" (UniqueName: \"kubernetes.io/projected/84914ee8-d996-409e-a212-d29119cc5de8-kube-api-access-4prd7\") pod \"payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p\" (UID: \"84914ee8-d996-409e-a212-d29119cc5de8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"
Nov 03 19:45:27.951916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.929227    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84914ee8-d996-409e-a212-d29119cc5de8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p\" (UID: \"84914ee8-d996-409e-a212-d29119cc5de8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"
Nov 03 19:45:27.951916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:27.930632    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7148f3-a2df-43bb-bf04-d5a0fddc1a13-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd\" (UID: \"fe7148f3-a2df-43bb-bf04-d5a0fddc1a13\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"
Nov 03 19:45:28.080860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:28.055265    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4prd7\" (UniqueName: \"kubernetes.io/projected/84914ee8-d996-409e-a212-d29119cc5de8-kube-api-access-4prd7\") pod \"payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p\" (UID: \"84914ee8-d996-409e-a212-d29119cc5de8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"
Nov 03 19:45:28.080860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:28.055335    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84914ee8-d996-409e-a212-d29119cc5de8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p\" (UID: \"84914ee8-d996-409e-a212-d29119cc5de8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"
Nov 03 19:45:28.080860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:28.059858    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84914ee8-d996-409e-a212-d29119cc5de8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p\" (UID: \"84914ee8-d996-409e-a212-d29119cc5de8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"
Nov 03 19:45:28.118927 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:28.101989    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:28.124523 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:28.124483    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwqkw\" (UniqueName: \"kubernetes.io/projected/fe7148f3-a2df-43bb-bf04-d5a0fddc1a13-kube-api-access-fwqkw\") pod \"payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd\" (UID: \"fe7148f3-a2df-43bb-bf04-d5a0fddc1a13\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"
Nov 03 19:45:28.155775 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:28.154751    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4prd7\" (UniqueName: \"kubernetes.io/projected/84914ee8-d996-409e-a212-d29119cc5de8-kube-api-access-4prd7\") pod \"payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p\" (UID: \"84914ee8-d996-409e-a212-d29119cc5de8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"
Nov 03 19:45:28.157097 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:28.156121    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:45:28.387986 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:28.385900    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"
Nov 03 19:45:28.390018 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:28.389803    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"
Nov 03 19:45:28.706138 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:28.706106    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"]
Nov 03 19:45:28.706946 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:45:28.706923    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84914ee8_d996_409e_a212_d29119cc5de8.slice/crio-e1fe30b5e857cb0db8e32005cf5c9ab5356fc38fad986cb05299fc84936ae8c3 WatchSource:0}: Error finding container e1fe30b5e857cb0db8e32005cf5c9ab5356fc38fad986cb05299fc84936ae8c3: Status 404 returned error can't find the container with id e1fe30b5e857cb0db8e32005cf5c9ab5356fc38fad986cb05299fc84936ae8c3
Nov 03 19:45:28.715299 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:45:28.715269    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7148f3_a2df_43bb_bf04_d5a0fddc1a13.slice/crio-61ed9f1946f95ce07b7e175fc173d7bc007bacf14c135d7351540c2f34868d7c WatchSource:0}: Error finding container 61ed9f1946f95ce07b7e175fc173d7bc007bacf14c135d7351540c2f34868d7c: Status 404 returned error can't find the container with id 61ed9f1946f95ce07b7e175fc173d7bc007bacf14c135d7351540c2f34868d7c
Nov 03 19:45:28.716713 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:28.716557    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"]
Nov 03 19:45:28.913607 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:28.913572    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:29.333427 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:29.333386    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd" event={"ID":"fe7148f3-a2df-43bb-bf04-d5a0fddc1a13","Type":"ContainerStarted","Data":"d8b770215c431b8c26e1e26cdd5842f6b8eac40fe94e42fd83844e6de351a13a"}
Nov 03 19:45:29.333632 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:29.333456    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd" event={"ID":"fe7148f3-a2df-43bb-bf04-d5a0fddc1a13","Type":"ContainerStarted","Data":"61ed9f1946f95ce07b7e175fc173d7bc007bacf14c135d7351540c2f34868d7c"}
Nov 03 19:45:29.334635 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:29.334610    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"
Nov 03 19:45:29.336777 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:29.336747    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p" event={"ID":"84914ee8-d996-409e-a212-d29119cc5de8","Type":"ContainerStarted","Data":"9f95679c81307d8103509a7e97ff0a493d8d1b9dd8efadbfbd02177cdf43a8a9"}
Nov 03 19:45:29.336915 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:29.336802    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p" event={"ID":"84914ee8-d996-409e-a212-d29119cc5de8","Type":"ContainerStarted","Data":"e1fe30b5e857cb0db8e32005cf5c9ab5356fc38fad986cb05299fc84936ae8c3"}
Nov 03 19:45:29.357567 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:29.357503    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd" podStartSLOduration=2.357483942 podStartE2EDuration="2.357483942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:29.355487366 +0000 UTC m=+1085.609794592" watchObservedRunningTime="2026-11-03 19:45:29.357483942 +0000 UTC m=+1085.611791168"
Nov 03 19:45:29.381628 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:29.381575    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p" podStartSLOduration=2.381558321 podStartE2EDuration="2.381558321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:29.377718605 +0000 UTC m=+1085.632025829" watchObservedRunningTime="2026-11-03 19:45:29.381558321 +0000 UTC m=+1085.635865546"
Nov 03 19:45:29.524784 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:29.524747    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:30.376225 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:30.376187    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:30.423363 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:30.423330    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"
Nov 03 19:45:30.516523 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:30.516373    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:30.657922 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:30.657744    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:31.252468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:31.252432    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:31.904329 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:31.904298    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:32.538971 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:32.531989    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:33.063951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.063913    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"]
Nov 03 19:45:33.070012 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.069983    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:45:33.077226 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.076956    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-2b2e60-serving-cert\""
Nov 03 19:45:33.080460 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.080426    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"]
Nov 03 19:45:33.112182 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.112152    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76b97\" (UniqueName: \"kubernetes.io/projected/17572c14-846c-4c5d-a141-6d36045da219-kube-api-access-76b97\") pod \"maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg\" (UID: \"17572c14-846c-4c5d-a141-6d36045da219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:45:33.112330 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.112272    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17572c14-846c-4c5d-a141-6d36045da219-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg\" (UID: \"17572c14-846c-4c5d-a141-6d36045da219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:45:33.112453 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.112432    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17572c14-846c-4c5d-a141-6d36045da219-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg\" (UID: \"17572c14-846c-4c5d-a141-6d36045da219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:45:33.112503 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.112482    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17572c14-846c-4c5d-a141-6d36045da219-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg\" (UID: \"17572c14-846c-4c5d-a141-6d36045da219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:45:33.169409 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.169245    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:33.236858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.219861    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76b97\" (UniqueName: \"kubernetes.io/projected/17572c14-846c-4c5d-a141-6d36045da219-kube-api-access-76b97\") pod \"maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg\" (UID: \"17572c14-846c-4c5d-a141-6d36045da219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:45:33.236858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.228605    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17572c14-846c-4c5d-a141-6d36045da219-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg\" (UID: \"17572c14-846c-4c5d-a141-6d36045da219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:45:33.236858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.228827    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17572c14-846c-4c5d-a141-6d36045da219-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg\" (UID: \"17572c14-846c-4c5d-a141-6d36045da219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:45:33.236858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.228949    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17572c14-846c-4c5d-a141-6d36045da219-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg\" (UID: \"17572c14-846c-4c5d-a141-6d36045da219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:45:33.238219 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.238189    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17572c14-846c-4c5d-a141-6d36045da219-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg\" (UID: \"17572c14-846c-4c5d-a141-6d36045da219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:45:33.245920 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.245890    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17572c14-846c-4c5d-a141-6d36045da219-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg\" (UID: \"17572c14-846c-4c5d-a141-6d36045da219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:45:33.257642 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.257543    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76b97\" (UniqueName: \"kubernetes.io/projected/17572c14-846c-4c5d-a141-6d36045da219-kube-api-access-76b97\") pod \"maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg\" (UID: \"17572c14-846c-4c5d-a141-6d36045da219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:45:33.270891 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.270641    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17572c14-846c-4c5d-a141-6d36045da219-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg\" (UID: \"17572c14-846c-4c5d-a141-6d36045da219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:45:33.326665 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.326569    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:33.388686 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.388650    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:45:33.475504 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.475343    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:33.904862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.897157    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:33.946215 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:33.946191    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"]
Nov 03 19:45:34.271907 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:34.270245    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:34.455860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:34.440367    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:34.607349 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:34.607282    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg" event={"ID":"17572c14-846c-4c5d-a141-6d36045da219","Type":"ContainerStarted","Data":"e96fdfd6c9b32d446625e572861d6386439ae6c5fe1bdf94066bcde0e896ed83"}
Nov 03 19:45:34.701764 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:34.701720    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:35.317722 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:35.317683    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:35.616239 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:35.616142    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg" event={"ID":"17572c14-846c-4c5d-a141-6d36045da219","Type":"ContainerStarted","Data":"7124099ad916ab7128e4a00c8f16b9ee2c2f9666c198dc8c6749407c2f5d95c1"}
Nov 03 19:45:35.616399 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:35.616284    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:45:35.644762 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:35.644690    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg" podStartSLOduration=1.969384835 podStartE2EDuration="2.644670889s" totalImagesPullingTime="675.286054ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:35.642111262 +0000 UTC m=+1091.896418488" watchObservedRunningTime="2026-11-03 19:45:35.644670889 +0000 UTC m=+1091.898978114"
Nov 03 19:45:35.930344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:35.930266    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:36.268077 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:36.267972    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:37.024628 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:37.024589    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:37.855789 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:37.855758    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:38.061227 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:38.061191    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:38.196113 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:38.196025    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:38.827918 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:38.827877    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:39.273239 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:39.273115    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:39.311388 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:39.311356    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"]
Nov 03 19:45:39.319996 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:39.319966    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"
Nov 03 19:45:39.332104 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:39.332074    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-64454a-dockercfg-bcfmd\""
Nov 03 19:45:39.333013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:39.332990    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"]
Nov 03 19:45:39.380760 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:39.380731    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-64454a\""
Nov 03 19:45:39.415213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:39.415121    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:39.488431 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:39.488393    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfhdk\" (UniqueName: \"kubernetes.io/projected/8d820e94-7b5a-48c6-8351-a45ea23de377-kube-api-access-qfhdk\") pod \"payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j\" (UID: \"8d820e94-7b5a-48c6-8351-a45ea23de377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"
Nov 03 19:45:39.488788 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:39.488764    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d820e94-7b5a-48c6-8351-a45ea23de377-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j\" (UID: \"8d820e94-7b5a-48c6-8351-a45ea23de377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"
Nov 03 19:45:39.603773 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:39.603695    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfhdk\" (UniqueName: \"kubernetes.io/projected/8d820e94-7b5a-48c6-8351-a45ea23de377-kube-api-access-qfhdk\") pod \"payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j\" (UID: \"8d820e94-7b5a-48c6-8351-a45ea23de377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"
Nov 03 19:45:39.606076 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:39.606047    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d820e94-7b5a-48c6-8351-a45ea23de377-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j\" (UID: \"8d820e94-7b5a-48c6-8351-a45ea23de377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"
Nov 03 19:45:39.607315 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:39.607111    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d820e94-7b5a-48c6-8351-a45ea23de377-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j\" (UID: \"8d820e94-7b5a-48c6-8351-a45ea23de377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"
Nov 03 19:45:39.630194 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:39.630148    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfhdk\" (UniqueName: \"kubernetes.io/projected/8d820e94-7b5a-48c6-8351-a45ea23de377-kube-api-access-qfhdk\") pod \"payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j\" (UID: \"8d820e94-7b5a-48c6-8351-a45ea23de377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"
Nov 03 19:45:39.642168 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:39.642131    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"
Nov 03 19:45:40.018094 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:40.017740    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:40.109874 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:40.108952    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"]
Nov 03 19:45:40.120301 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:45:40.120264    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d820e94_7b5a_48c6_8351_a45ea23de377.slice/crio-a23d183ec77f69ad634758d9a1fda837790fd80def9752eafd7830703aa4a5f7 WatchSource:0}: Error finding container a23d183ec77f69ad634758d9a1fda837790fd80def9752eafd7830703aa4a5f7: Status 404 returned error can't find the container with id a23d183ec77f69ad634758d9a1fda837790fd80def9752eafd7830703aa4a5f7
Nov 03 19:45:40.667414 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:40.667375    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j" event={"ID":"8d820e94-7b5a-48c6-8351-a45ea23de377","Type":"ContainerStarted","Data":"47880980c933540001ef68c5ce806efcd1c6481541c9dbb56962b7ea289f3b22"}
Nov 03 19:45:40.667880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:40.667433    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j" event={"ID":"8d820e94-7b5a-48c6-8351-a45ea23de377","Type":"ContainerStarted","Data":"a23d183ec77f69ad634758d9a1fda837790fd80def9752eafd7830703aa4a5f7"}
Nov 03 19:45:40.667880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:40.667469    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"
Nov 03 19:45:40.694079 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:40.694021    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j" podStartSLOduration=1.694002488 podStartE2EDuration="1.694002488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:40.689551494 +0000 UTC m=+1096.943858721" watchObservedRunningTime="2026-11-03 19:45:40.694002488 +0000 UTC m=+1096.948309743"
Nov 03 19:45:40.937559 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:40.936992    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:41.407863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:41.407807    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"
Nov 03 19:45:41.408040 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:41.407907    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"
Nov 03 19:45:41.645879 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:41.626200    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:41.663651 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:41.662726    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:45:42.342220 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:42.336779    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:42.864749 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:42.864698    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:43.037803 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:43.037767    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:43.198907 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:43.198046    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:43.390892 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:43.387606    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:43.425746 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:45:43.425494    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:45:43.991861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:43.977715    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:44.934445 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:44.934415    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:45.620945 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:45.620909    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:46.248888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.248818    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"]
Nov 03 19:45:46.254531 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.254504    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:45:46.262722 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.262690    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-64454a-serving-cert\""
Nov 03 19:45:46.265360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.265334    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"]
Nov 03 19:45:46.330574 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.330537    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:46.428070 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.428032    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b3d450-df16-4c35-8095-4d0640c7a422-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24\" (UID: \"98b3d450-df16-4c35-8095-4d0640c7a422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:45:46.428254 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.428152    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98b3d450-df16-4c35-8095-4d0640c7a422-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24\" (UID: \"98b3d450-df16-4c35-8095-4d0640c7a422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:45:46.428254 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.428247    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj48n\" (UniqueName: \"kubernetes.io/projected/98b3d450-df16-4c35-8095-4d0640c7a422-kube-api-access-fj48n\") pod \"maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24\" (UID: \"98b3d450-df16-4c35-8095-4d0640c7a422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:45:46.428458 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.428427    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98b3d450-df16-4c35-8095-4d0640c7a422-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24\" (UID: \"98b3d450-df16-4c35-8095-4d0640c7a422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:45:46.458689 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.458653    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:46.539911 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.531448    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj48n\" (UniqueName: \"kubernetes.io/projected/98b3d450-df16-4c35-8095-4d0640c7a422-kube-api-access-fj48n\") pod \"maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24\" (UID: \"98b3d450-df16-4c35-8095-4d0640c7a422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:45:46.539911 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.531692    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98b3d450-df16-4c35-8095-4d0640c7a422-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24\" (UID: \"98b3d450-df16-4c35-8095-4d0640c7a422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:45:46.539911 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.531891    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b3d450-df16-4c35-8095-4d0640c7a422-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24\" (UID: \"98b3d450-df16-4c35-8095-4d0640c7a422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:45:46.539911 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.532008    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98b3d450-df16-4c35-8095-4d0640c7a422-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24\" (UID: \"98b3d450-df16-4c35-8095-4d0640c7a422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:45:46.539911 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.534349    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b3d450-df16-4c35-8095-4d0640c7a422-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24\" (UID: \"98b3d450-df16-4c35-8095-4d0640c7a422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:45:46.539911 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.538871    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98b3d450-df16-4c35-8095-4d0640c7a422-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24\" (UID: \"98b3d450-df16-4c35-8095-4d0640c7a422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:45:46.560203 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.560134    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98b3d450-df16-4c35-8095-4d0640c7a422-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24\" (UID: \"98b3d450-df16-4c35-8095-4d0640c7a422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:45:46.567811 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.566728    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj48n\" (UniqueName: \"kubernetes.io/projected/98b3d450-df16-4c35-8095-4d0640c7a422-kube-api-access-fj48n\") pod \"maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24\" (UID: \"98b3d450-df16-4c35-8095-4d0640c7a422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:45:46.583898 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:46.579204    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:45:47.229354 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:47.228589    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"]
Nov 03 19:45:47.311849 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:45:47.304562    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b3d450_df16_4c35_8095_4d0640c7a422.slice/crio-df0f6d396e961b7ca688e6fafe1299e4a4cf2c5b7816549bce6573f976fb7d8f WatchSource:0}: Error finding container df0f6d396e961b7ca688e6fafe1299e4a4cf2c5b7816549bce6573f976fb7d8f: Status 404 returned error can't find the container with id df0f6d396e961b7ca688e6fafe1299e4a4cf2c5b7816549bce6573f976fb7d8f
Nov 03 19:45:47.515918 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:47.500767    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7104-f39be0"]
Nov 03 19:45:47.531865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:47.516750    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:47.592144 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:47.592115    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7104-f39be0"]
Nov 03 19:45:47.592477 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:47.592431    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-f39be0"
Nov 03 19:45:47.604744 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:47.604715    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hjqcx\""
Nov 03 19:45:47.705992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:47.705951    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqwmg\" (UniqueName: \"kubernetes.io/projected/bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990-kube-api-access-rqwmg\") pod \"test-curl-7104-f39be0\" (UID: \"bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990\") " pod="opendatahub/test-curl-7104-f39be0"
Nov 03 19:45:47.813494 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:47.812282    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqwmg\" (UniqueName: \"kubernetes.io/projected/bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990-kube-api-access-rqwmg\") pod \"test-curl-7104-f39be0\" (UID: \"bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990\") " pod="opendatahub/test-curl-7104-f39be0"
Nov 03 19:45:47.836708 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:47.836651    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqwmg\" (UniqueName: \"kubernetes.io/projected/bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990-kube-api-access-rqwmg\") pod \"test-curl-7104-f39be0\" (UID: \"bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990\") " pod="opendatahub/test-curl-7104-f39be0"
Nov 03 19:45:47.967969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:47.967927    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24" event={"ID":"98b3d450-df16-4c35-8095-4d0640c7a422","Type":"ContainerStarted","Data":"5681ee443c0c567d40acc1b7926be71ffa54ce037b8dfd29ef4fc78311e15898"}
Nov 03 19:45:47.968169 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:47.967990    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24" event={"ID":"98b3d450-df16-4c35-8095-4d0640c7a422","Type":"ContainerStarted","Data":"df0f6d396e961b7ca688e6fafe1299e4a4cf2c5b7816549bce6573f976fb7d8f"}
Nov 03 19:45:47.968169 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:47.968062    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:45:47.991871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:47.991804    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24" podStartSLOduration=1.448737919 podStartE2EDuration="1.991789145s" totalImagesPullingTime="543.051226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:47.99114281 +0000 UTC m=+1104.245450164" watchObservedRunningTime="2026-11-03 19:45:47.991789145 +0000 UTC m=+1104.246096371"
Nov 03 19:45:48.019600 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:48.019573    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-f39be0"
Nov 03 19:45:48.232895 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:48.222952    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:48.325402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:48.325309    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7104-f39be0"]
Nov 03 19:45:48.330938 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:45:48.330909    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb2fc12a_00f3_46a4_9ccc_8d8d17a1f990.slice/crio-e6dc53000a9f7a91d7830b493bfb0afad955bcc2e78088fb0a7161edeebf6392 WatchSource:0}: Error finding container e6dc53000a9f7a91d7830b493bfb0afad955bcc2e78088fb0a7161edeebf6392: Status 404 returned error can't find the container with id e6dc53000a9f7a91d7830b493bfb0afad955bcc2e78088fb0a7161edeebf6392
Nov 03 19:45:48.829680 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:48.829639    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:48.980498 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:48.980456    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7104-f39be0" event={"ID":"bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990","Type":"ContainerStarted","Data":"da477ce0de3c927147ef6b7c8cc794cb731a8cc0d4f5153be86f7c2160d5f77b"}
Nov 03 19:45:48.980674 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:48.980511    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7104-f39be0" event={"ID":"bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990","Type":"ContainerStarted","Data":"e6dc53000a9f7a91d7830b493bfb0afad955bcc2e78088fb0a7161edeebf6392"}
Nov 03 19:45:49.000043 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:48.999993    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7104-f39be0" podStartSLOduration=1.9999794899999999 podStartE2EDuration="1.99997949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:48.996395688 +0000 UTC m=+1105.250702912" watchObservedRunningTime="2026-11-03 19:45:48.99997949 +0000 UTC m=+1105.254286737"
Nov 03 19:45:49.569868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:49.562221    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:49.847873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:49.847762    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7104-f39be0"]
Nov 03 19:45:49.852617 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:49.852591    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7104-f39be0"]
Nov 03 19:45:50.021392 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:50.021358    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7104-aee98a"]
Nov 03 19:45:50.022046 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:50.022029    5237 cpu_manager.go:414] "containerMap: removing container" podUID="bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990" containerName="test-curl-7104-f39be0"
Nov 03 19:45:50.022108 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:50.022047    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990" containerName="test-curl-7104-f39be0"
Nov 03 19:45:50.022175 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:50.022165    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990" containerName="test-curl-7104-f39be0"
Nov 03 19:45:50.025729 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:50.025712    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-aee98a"
Nov 03 19:45:50.031533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:50.031498    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7104-aee98a"]
Nov 03 19:45:50.065149 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:50.065114    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9fqc\" (UniqueName: \"kubernetes.io/projected/10875911-d753-49a3-8106-6aadba8947d9-kube-api-access-q9fqc\") pod \"test-curl-7104-aee98a\" (UID: \"10875911-d753-49a3-8106-6aadba8947d9\") " pod="opendatahub/test-curl-7104-aee98a"
Nov 03 19:45:50.166701 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:50.166616    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9fqc\" (UniqueName: \"kubernetes.io/projected/10875911-d753-49a3-8106-6aadba8947d9-kube-api-access-q9fqc\") pod \"test-curl-7104-aee98a\" (UID: \"10875911-d753-49a3-8106-6aadba8947d9\") " pod="opendatahub/test-curl-7104-aee98a"
Nov 03 19:45:50.179669 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:50.179637    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9fqc\" (UniqueName: \"kubernetes.io/projected/10875911-d753-49a3-8106-6aadba8947d9-kube-api-access-q9fqc\") pod \"test-curl-7104-aee98a\" (UID: \"10875911-d753-49a3-8106-6aadba8947d9\") " pod="opendatahub/test-curl-7104-aee98a"
Nov 03 19:45:50.218744 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:50.218702    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:50.360907 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:50.356243    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-aee98a"
Nov 03 19:45:50.370415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:50.367103    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:50.731295 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:50.731205    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7104-aee98a"]
Nov 03 19:45:50.735508 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:45:50.735481    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10875911_d753_49a3_8106_6aadba8947d9.slice/crio-555356ad841cb8b27efa8fc343db5e87f5e5adaaaca2b13798cc948cd44c93c7 WatchSource:0}: Error finding container 555356ad841cb8b27efa8fc343db5e87f5e5adaaaca2b13798cc948cd44c93c7: Status 404 returned error can't find the container with id 555356ad841cb8b27efa8fc343db5e87f5e5adaaaca2b13798cc948cd44c93c7
Nov 03 19:45:50.996575 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:50.996482    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7104-aee98a" event={"ID":"10875911-d753-49a3-8106-6aadba8947d9","Type":"ContainerStarted","Data":"5ac73ac3da44452cf7b2bacc95354f5ab8e23eba4248eb773907a7229fe25080"}
Nov 03 19:45:50.996575 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:50.996532    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7104-aee98a" event={"ID":"10875911-d753-49a3-8106-6aadba8947d9","Type":"ContainerStarted","Data":"555356ad841cb8b27efa8fc343db5e87f5e5adaaaca2b13798cc948cd44c93c7"}
Nov 03 19:45:50.996575 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:50.996529    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7104-f39be0" podUID="bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990" containerName="test-curl-7104-f39be0" containerID="cri-o://da477ce0de3c927147ef6b7c8cc794cb731a8cc0d4f5153be86f7c2160d5f77b" gracePeriod=2
Nov 03 19:45:51.018958 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:51.018889    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7104-aee98a" podStartSLOduration=1.018871593 podStartE2EDuration="1.018871593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:51.018733965 +0000 UTC m=+1107.273041192" watchObservedRunningTime="2026-11-03 19:45:51.018871593 +0000 UTC m=+1107.273178823"
Nov 03 19:45:51.131264 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:51.131224    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:51.289397 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:51.289361    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:51.747765 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:51.747666    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"
Nov 03 19:45:51.786377 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:51.786341    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7104-aee98a"]
Nov 03 19:45:51.789797 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:51.789768    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:51.793384 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:51.791949    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7104-aee98a"]
Nov 03 19:45:51.960464 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:51.960421    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:52.032916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:52.027336    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7104-ef1d41"]
Nov 03 19:45:52.032916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:52.028351    5237 cpu_manager.go:414] "containerMap: removing container" podUID="10875911-d753-49a3-8106-6aadba8947d9" containerName="test-curl-7104-aee98a"
Nov 03 19:45:52.032916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:52.028373    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10875911-d753-49a3-8106-6aadba8947d9" containerName="test-curl-7104-aee98a"
Nov 03 19:45:52.032916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:52.028584    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="10875911-d753-49a3-8106-6aadba8947d9" containerName="test-curl-7104-aee98a"
Nov 03 19:45:52.150165 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:52.150134    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7104-ef1d41"]
Nov 03 19:45:52.150533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:52.150468    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-ef1d41"
Nov 03 19:45:52.248656 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:52.248615    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttqdx\" (UniqueName: \"kubernetes.io/projected/b7982894-7d28-4099-b630-a441af0c9707-kube-api-access-ttqdx\") pod \"test-curl-7104-ef1d41\" (UID: \"b7982894-7d28-4099-b630-a441af0c9707\") " pod="opendatahub/test-curl-7104-ef1d41"
Nov 03 19:45:52.369296 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:52.369144    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttqdx\" (UniqueName: \"kubernetes.io/projected/b7982894-7d28-4099-b630-a441af0c9707-kube-api-access-ttqdx\") pod \"test-curl-7104-ef1d41\" (UID: \"b7982894-7d28-4099-b630-a441af0c9707\") " pod="opendatahub/test-curl-7104-ef1d41"
Nov 03 19:45:52.395558 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:52.391410    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttqdx\" (UniqueName: \"kubernetes.io/projected/b7982894-7d28-4099-b630-a441af0c9707-kube-api-access-ttqdx\") pod \"test-curl-7104-ef1d41\" (UID: \"b7982894-7d28-4099-b630-a441af0c9707\") " pod="opendatahub/test-curl-7104-ef1d41"
Nov 03 19:45:52.633322 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:52.633284    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-ef1d41"
Nov 03 19:45:52.746734 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:52.746698    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:53.014567 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:53.014524    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7104-ef1d41"]
Nov 03 19:45:53.146618 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:53.146583    5237 generic.go:356] "Generic (PLEG): container finished" podID="bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990" containerID="da477ce0de3c927147ef6b7c8cc794cb731a8cc0d4f5153be86f7c2160d5f77b" exitCode=137
Nov 03 19:45:53.149448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:53.149184    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7104-ef1d41" event={"ID":"b7982894-7d28-4099-b630-a441af0c9707","Type":"ContainerStarted","Data":"aeaa088c6d21cf2ac41263764ede202765c3c50f372e1aabcb18146a2c7660ed"}
Nov 03 19:45:53.149448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:53.149298    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7104-aee98a" podUID="10875911-d753-49a3-8106-6aadba8947d9" containerName="test-curl-7104-aee98a" containerID="cri-o://5ac73ac3da44452cf7b2bacc95354f5ab8e23eba4248eb773907a7229fe25080" gracePeriod=2
Nov 03 19:45:53.275928 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:53.275903    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-f39be0"
Nov 03 19:45:53.399581 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:53.399277    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990-kube-api-access-rqwmg\" (UniqueName: \"kubernetes.io/projected/bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990-kube-api-access-rqwmg\") pod \"bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990\" (UID: \"bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990\") "
Nov 03 19:45:53.401922 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:53.401883    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990-kube-api-access-rqwmg" pod "bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990" (UID: "bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990"). InnerVolumeSpecName "kube-api-access-rqwmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:53.500904 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:53.500869    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqwmg\" (UniqueName: \"kubernetes.io/projected/bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990-kube-api-access-rqwmg\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:53.562868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:53.562578    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:53.863525 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:53.863451    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990" path="/var/lib/kubelet/pods/bb2fc12a-00f3-46a4-9ccc-8d8d17a1f990/volumes"
Nov 03 19:45:53.987434 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:53.987401    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:45:54.156560 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:54.156480    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-f39be0"
Nov 03 19:45:54.156560 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:54.156504    5237 scope.go:122] "RemoveContainer" containerID="da477ce0de3c927147ef6b7c8cc794cb731a8cc0d4f5153be86f7c2160d5f77b"
Nov 03 19:45:54.158308 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:54.158280    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7104-ef1d41" event={"ID":"b7982894-7d28-4099-b630-a441af0c9707","Type":"ContainerStarted","Data":"353d7adccbfccd95dc7d24c23d15b7021ec89836eccdec80c859d66064cd2fd1"}
Nov 03 19:45:54.178028 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:54.177974    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7104-ef1d41" podStartSLOduration=3.177959168 podStartE2EDuration="3.177959168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:54.174897664 +0000 UTC m=+1110.429204892" watchObservedRunningTime="2026-11-03 19:45:54.177959168 +0000 UTC m=+1110.432266416"
Nov 03 19:45:54.511576 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:54.511491    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:55.041829 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.041795    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7104-ef1d41"]
Nov 03 19:45:55.047073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.047036    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7104-ef1d41"]
Nov 03 19:45:55.205969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.205940    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7104-4d9fff"]
Nov 03 19:45:55.206652 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.206635    5237 cpu_manager.go:414] "containerMap: removing container" podUID="b7982894-7d28-4099-b630-a441af0c9707" containerName="test-curl-7104-ef1d41"
Nov 03 19:45:55.206749 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.206653    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7982894-7d28-4099-b630-a441af0c9707" containerName="test-curl-7104-ef1d41"
Nov 03 19:45:55.206803 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.206792    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7982894-7d28-4099-b630-a441af0c9707" containerName="test-curl-7104-ef1d41"
Nov 03 19:45:55.212424 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.212399    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-4d9fff"
Nov 03 19:45:55.220973 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.220943    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7104-4d9fff"]
Nov 03 19:45:55.279607 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.279564    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64tlx\" (UniqueName: \"kubernetes.io/projected/e9fe734e-8435-454e-8752-75fe14d62210-kube-api-access-64tlx\") pod \"test-curl-7104-4d9fff\" (UID: \"e9fe734e-8435-454e-8752-75fe14d62210\") " pod="opendatahub/test-curl-7104-4d9fff"
Nov 03 19:45:55.311357 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.311258    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:55.371521 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.371495    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-aee98a"
Nov 03 19:45:55.380696 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.380668    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64tlx\" (UniqueName: \"kubernetes.io/projected/e9fe734e-8435-454e-8752-75fe14d62210-kube-api-access-64tlx\") pod \"test-curl-7104-4d9fff\" (UID: \"e9fe734e-8435-454e-8752-75fe14d62210\") " pod="opendatahub/test-curl-7104-4d9fff"
Nov 03 19:45:55.396291 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.396254    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64tlx\" (UniqueName: \"kubernetes.io/projected/e9fe734e-8435-454e-8752-75fe14d62210-kube-api-access-64tlx\") pod \"test-curl-7104-4d9fff\" (UID: \"e9fe734e-8435-454e-8752-75fe14d62210\") " pod="opendatahub/test-curl-7104-4d9fff"
Nov 03 19:45:55.458826 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.458789    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:55.481854 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.481817    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10875911-d753-49a3-8106-6aadba8947d9-kube-api-access-q9fqc\" (UniqueName: \"kubernetes.io/projected/10875911-d753-49a3-8106-6aadba8947d9-kube-api-access-q9fqc\") pod \"10875911-d753-49a3-8106-6aadba8947d9\" (UID: \"10875911-d753-49a3-8106-6aadba8947d9\") "
Nov 03 19:45:55.484059 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.484026    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10875911-d753-49a3-8106-6aadba8947d9-kube-api-access-q9fqc" pod "10875911-d753-49a3-8106-6aadba8947d9" (UID: "10875911-d753-49a3-8106-6aadba8947d9"). InnerVolumeSpecName "kube-api-access-q9fqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:55.537497 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.537464    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-4d9fff"
Nov 03 19:45:55.584541 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.584456    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9fqc\" (UniqueName: \"kubernetes.io/projected/10875911-d753-49a3-8106-6aadba8947d9-kube-api-access-q9fqc\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:55.679482 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.679442    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:55.837919 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:55.837219    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7104-4d9fff"]
Nov 03 19:45:56.022868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:56.022722    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10875911-d753-49a3-8106-6aadba8947d9" path="/var/lib/kubelet/pods/10875911-d753-49a3-8106-6aadba8947d9/volumes"
Nov 03 19:45:56.239792 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:56.239740    5237 generic.go:356] "Generic (PLEG): container finished" podID="10875911-d753-49a3-8106-6aadba8947d9" containerID="5ac73ac3da44452cf7b2bacc95354f5ab8e23eba4248eb773907a7229fe25080" exitCode=137
Nov 03 19:45:56.240303 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:56.239892    5237 scope.go:122] "RemoveContainer" containerID="5ac73ac3da44452cf7b2bacc95354f5ab8e23eba4248eb773907a7229fe25080"
Nov 03 19:45:56.240303 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:56.240085    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-aee98a"
Nov 03 19:45:56.249033 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:56.248390    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7104-ef1d41" podUID="b7982894-7d28-4099-b630-a441af0c9707" containerName="test-curl-7104-ef1d41" containerID="cri-o://353d7adccbfccd95dc7d24c23d15b7021ec89836eccdec80c859d66064cd2fd1" gracePeriod=2
Nov 03 19:45:56.249033 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:56.248732    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7104-4d9fff" event={"ID":"e9fe734e-8435-454e-8752-75fe14d62210","Type":"ContainerStarted","Data":"06b66c04a6ff07cae30e4babee531a8f262b4ce87186eb437485b6b280dad985"}
Nov 03 19:45:56.304432 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:56.303120    5237 scope.go:122] "RemoveContainer" containerID="5ac73ac3da44452cf7b2bacc95354f5ab8e23eba4248eb773907a7229fe25080"
Nov 03 19:45:56.304809 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:45:56.304705    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac73ac3da44452cf7b2bacc95354f5ab8e23eba4248eb773907a7229fe25080\": container with ID starting with 5ac73ac3da44452cf7b2bacc95354f5ab8e23eba4248eb773907a7229fe25080 not found: ID does not exist" containerID="5ac73ac3da44452cf7b2bacc95354f5ab8e23eba4248eb773907a7229fe25080"
Nov 03 19:45:56.304809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:56.304746    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac73ac3da44452cf7b2bacc95354f5ab8e23eba4248eb773907a7229fe25080"} err="failed to get container status \"5ac73ac3da44452cf7b2bacc95354f5ab8e23eba4248eb773907a7229fe25080\": rpc error: code = NotFound desc = could not find container \"5ac73ac3da44452cf7b2bacc95354f5ab8e23eba4248eb773907a7229fe25080\": container with ID starting with 5ac73ac3da44452cf7b2bacc95354f5ab8e23eba4248eb773907a7229fe25080 not found: ID does not exist"
Nov 03 19:45:56.693866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:56.682853    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:56.825551 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:56.825467    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:57.658779 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:57.648169    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:57.850102 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:57.850018    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7104-4d9fff" event={"ID":"e9fe734e-8435-454e-8752-75fe14d62210","Type":"ContainerStarted","Data":"9307605f13e1ae731172d472c1941462a83c6edaa5ba5d8bd39951050ae35243"}
Nov 03 19:45:57.926687 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:57.921464    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7104-4d9fff" podStartSLOduration=2.921444195 podStartE2EDuration="2.921444195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:57.921228616 +0000 UTC m=+1114.175535842" watchObservedRunningTime="2026-11-03 19:45:57.921444195 +0000 UTC m=+1114.175751420"
Nov 03 19:45:57.963181 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:57.963112    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.262380 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:59.261058    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.293965 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:59.266933    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 19:45:59.303425 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:59.301935    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:45:59.327363 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:59.319007    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.369890 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:59.362771    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.369890 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:59.363154    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 19:45:59.390450 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:59.388423    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:59.629858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:59.628039    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.629858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:59.629015    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.876752 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:59.876186    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.880131 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:45:59.880039    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.088876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.082291    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 19:46:00.088876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.082649    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:46:00.146925 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.141887    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.203138 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.203097    5237 generic.go:356] "Generic (PLEG): container finished" podID="b7982894-7d28-4099-b630-a441af0c9707" containerID="353d7adccbfccd95dc7d24c23d15b7021ec89836eccdec80c859d66064cd2fd1" exitCode=137
Nov 03 19:46:00.212990 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.211007    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-ef1d41"
Nov 03 19:46:00.238984 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.238941    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.322875 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.321640    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.322875 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.322300    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7982894-7d28-4099-b630-a441af0c9707-kube-api-access-ttqdx\" (UniqueName: \"kubernetes.io/projected/b7982894-7d28-4099-b630-a441af0c9707-kube-api-access-ttqdx\") pod \"b7982894-7d28-4099-b630-a441af0c9707\" (UID: \"b7982894-7d28-4099-b630-a441af0c9707\") "
Nov 03 19:46:00.337514 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.337433    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.338698 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.338380    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:46:00.338698 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.338454    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:46:00.430173 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.413951    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.430173 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.414566    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:46:00.725933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.592904    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7982894-7d28-4099-b630-a441af0c9707-kube-api-access-ttqdx" pod "b7982894-7d28-4099-b630-a441af0c9707" (UID: "b7982894-7d28-4099-b630-a441af0c9707"). InnerVolumeSpecName "kube-api-access-ttqdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:00.725933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.650937    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttqdx\" (UniqueName: \"kubernetes.io/projected/b7982894-7d28-4099-b630-a441af0c9707-kube-api-access-ttqdx\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:00.725933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.654098    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 19:46:00.725933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.654597    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:46:00.870015 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:00.862583    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded"
Nov 03 19:46:01.138942 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:01.125305    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:46:01.154183 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:01.154128    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:01.154810 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:01.154683    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:46:01.219262 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:01.219233    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:46:01.222952 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:01.222791    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:46:01.222952 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:01.222860    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:46:01.222952 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:01.222891    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:46:01.257771 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:01.253308    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-ef1d41"
Nov 03 19:46:01.257771 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:01.253359    5237 scope.go:122] "RemoveContainer" containerID="353d7adccbfccd95dc7d24c23d15b7021ec89836eccdec80c859d66064cd2fd1"
Nov 03 19:46:01.400915 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:46:01.399251    5237 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7104-ef1d41\" is forbidden: User \"system:node:ip-10-0-133-29.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-29.us-east-2.compute.internal' and this object" podUID="b7982894-7d28-4099-b630-a441af0c9707" pod="opendatahub/test-curl-7104-ef1d41"
Nov 03 19:46:01.400915 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:01.399309    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7104-4d9fff"]
Nov 03 19:46:01.400915 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:01.399512    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7104-4d9fff" podUID="e9fe734e-8435-454e-8752-75fe14d62210" containerName="test-curl-7104-4d9fff" containerID="cri-o://9307605f13e1ae731172d472c1941462a83c6edaa5ba5d8bd39951050ae35243" gracePeriod=2
Nov 03 19:46:01.412925 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:01.412543    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:01.416886 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:46:01.416820    5237 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7104-4d9fff\" is forbidden: User \"system:node:ip-10-0-133-29.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-29.us-east-2.compute.internal' and this object" podUID="e9fe734e-8435-454e-8752-75fe14d62210" pod="opendatahub/test-curl-7104-4d9fff"
Nov 03 19:46:01.419453 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:46:01.419421    5237 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7104-ef1d41\" is forbidden: User \"system:node:ip-10-0-133-29.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-29.us-east-2.compute.internal' and this object" podUID="b7982894-7d28-4099-b630-a441af0c9707" pod="opendatahub/test-curl-7104-ef1d41"
Nov 03 19:46:01.419532 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:01.419471    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7104-4d9fff"]
Nov 03 19:46:01.862059 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:01.862021    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7982894-7d28-4099-b630-a441af0c9707" path="/var/lib/kubelet/pods/b7982894-7d28-4099-b630-a441af0c9707/volumes"
Nov 03 19:46:02.192913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:02.192527    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:02.643648 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:02.643620    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:02.847066 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:02.847024    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:03.163324 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:03.163008    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:03.433313 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:03.433277    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:03.568460 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:03.568424    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:03.773117 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:03.773092    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-4d9fff"
Nov 03 19:46:03.932415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:03.932383    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9fe734e-8435-454e-8752-75fe14d62210-kube-api-access-64tlx\" (UniqueName: \"kubernetes.io/projected/e9fe734e-8435-454e-8752-75fe14d62210-kube-api-access-64tlx\") pod \"e9fe734e-8435-454e-8752-75fe14d62210\" (UID: \"e9fe734e-8435-454e-8752-75fe14d62210\") "
Nov 03 19:46:03.934510 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:03.934478    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9fe734e-8435-454e-8752-75fe14d62210-kube-api-access-64tlx" pod "e9fe734e-8435-454e-8752-75fe14d62210" (UID: "e9fe734e-8435-454e-8752-75fe14d62210"). InnerVolumeSpecName "kube-api-access-64tlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:03.958649 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:03.958618    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:04.035134 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:04.035107    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64tlx\" (UniqueName: \"kubernetes.io/projected/e9fe734e-8435-454e-8752-75fe14d62210-kube-api-access-64tlx\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:04.124018 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:04.123981    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:04.350608 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:04.350521    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:04.517868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:04.512978    5237 generic.go:356] "Generic (PLEG): container finished" podID="e9fe734e-8435-454e-8752-75fe14d62210" containerID="9307605f13e1ae731172d472c1941462a83c6edaa5ba5d8bd39951050ae35243" exitCode=137
Nov 03 19:46:04.517868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:04.513335    5237 scope.go:122] "RemoveContainer" containerID="9307605f13e1ae731172d472c1941462a83c6edaa5ba5d8bd39951050ae35243"
Nov 03 19:46:04.517868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:04.513593    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-4d9fff"
Nov 03 19:46:04.574887 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:04.574850    5237 scope.go:122] "RemoveContainer" containerID="9307605f13e1ae731172d472c1941462a83c6edaa5ba5d8bd39951050ae35243"
Nov 03 19:46:04.599468 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:46:04.599433    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9307605f13e1ae731172d472c1941462a83c6edaa5ba5d8bd39951050ae35243\": container with ID starting with 9307605f13e1ae731172d472c1941462a83c6edaa5ba5d8bd39951050ae35243 not found: ID does not exist" containerID="9307605f13e1ae731172d472c1941462a83c6edaa5ba5d8bd39951050ae35243"
Nov 03 19:46:04.600001 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:04.599748    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9307605f13e1ae731172d472c1941462a83c6edaa5ba5d8bd39951050ae35243"} err="failed to get container status \"9307605f13e1ae731172d472c1941462a83c6edaa5ba5d8bd39951050ae35243\": rpc error: code = NotFound desc = could not find container \"9307605f13e1ae731172d472c1941462a83c6edaa5ba5d8bd39951050ae35243\": container with ID starting with 9307605f13e1ae731172d472c1941462a83c6edaa5ba5d8bd39951050ae35243 not found: ID does not exist"
Nov 03 19:46:05.179655 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:05.179556    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:05.863567 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:05.863531    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9fe734e-8435-454e-8752-75fe14d62210" path="/var/lib/kubelet/pods/e9fe734e-8435-454e-8752-75fe14d62210/volumes"
Nov 03 19:46:05.961872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:05.958979    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:06.656865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:06.656808    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:07.345257 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:07.345219    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:08.328136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:08.328040    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:08.487859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:08.481549    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:08.551013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:08.550979    5237 generic.go:356] "Generic (PLEG): container finished" podID="68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" containerID="7ec9b427e11b45b9ede06afc80f5789fcf3b67c99b2780c46cb5e4d248623663" exitCode=0
Nov 03 19:46:08.551202 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:08.551047    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6" event={"ID":"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a","Type":"ContainerDied","Data":"7ec9b427e11b45b9ede06afc80f5789fcf3b67c99b2780c46cb5e4d248623663"}
Nov 03 19:46:08.551590 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:08.551571    5237 scope.go:122] "RemoveContainer" containerID="7ec9b427e11b45b9ede06afc80f5789fcf3b67c99b2780c46cb5e4d248623663"
Nov 03 19:46:08.553550 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:08.553516    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:46:08.553626 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:08.553610    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:46:08.553666 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:08.553656    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:46:08.644948 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:08.644908    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:09.190236 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:09.188907    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:09.616859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:09.616250    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6" event={"ID":"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a","Type":"ContainerStarted","Data":"f12187738d358a21d868f3d8e9c782a398744f7cc12321fb5c36190767c7a186"}
Nov 03 19:46:09.694721 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:09.694192    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:46:09.694721 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:09.694326    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:46:09.694721 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:09.694370    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6" podUID="68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:46:10.150090 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:10.127875    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:10.693313 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:10.693269    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:46:10.693913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:10.693357    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6" podUID="68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:46:10.858480 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:10.858440    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:11.695004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:11.694882    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:46:11.695004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:11.694979    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6" podUID="68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:46:11.742389 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:11.742352    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:46:11.819857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:11.813054    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:12.439548 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:12.439507    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:12.693396 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:12.693299    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:46:12.693396 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:12.693370    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6" podUID="68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:46:13.117044 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.117005    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:13.319148 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:46:13.319080    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:46:13.349697 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.349657    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"]
Nov 03 19:46:13.350560 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.350533    5237 cpu_manager.go:414] "containerMap: removing container" podUID="e9fe734e-8435-454e-8752-75fe14d62210" containerName="test-curl-7104-4d9fff"
Nov 03 19:46:13.350560 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.350556    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9fe734e-8435-454e-8752-75fe14d62210" containerName="test-curl-7104-4d9fff"
Nov 03 19:46:13.350720 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.350645    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9fe734e-8435-454e-8752-75fe14d62210" containerName="test-curl-7104-4d9fff"
Nov 03 19:46:13.354226 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.354207    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.358266 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.358239    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-c54f85-openshift-default-dockercfg-w8jpj\""
Nov 03 19:46:13.379913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.379654    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"]
Nov 03 19:46:13.505202 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.505165    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e6fabb-7410-46bc-a67f-40d9116f02eb-istiod-ca-cert\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.505464 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.505231    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr6tk\" (UniqueName: \"kubernetes.io/projected/92e6fabb-7410-46bc-a67f-40d9116f02eb-kube-api-access-zr6tk\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.505464 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.505336    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-envoy\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.505464 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.505377    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-podinfo\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.505620 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.505511    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-token\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.505620 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.505535    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-workload-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.505620 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.505552    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-workload-certs\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.505620 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.505566    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-data\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.505903 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.505876    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-credential-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.607145 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.607095    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-token\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.607145 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.607154    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-workload-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.607425 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.607179    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-workload-certs\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.607425 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.607202    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-data\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.607559 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.607526    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-credential-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.607685 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.607658    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-workload-certs\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.607775 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.607692    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e6fabb-7410-46bc-a67f-40d9116f02eb-istiod-ca-cert\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.607775 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.607670    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-workload-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.607775 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.607755    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr6tk\" (UniqueName: \"kubernetes.io/projected/92e6fabb-7410-46bc-a67f-40d9116f02eb-kube-api-access-zr6tk\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.607961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.607765    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-data\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.607961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.607801    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-envoy\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.607961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.607830    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-podinfo\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.607961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.607924    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-credential-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.608486 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.608461    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e6fabb-7410-46bc-a67f-40d9116f02eb-istiod-ca-cert\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.610128 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.610099    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-envoy\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.610392 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.610375    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-podinfo\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.619786 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.619762    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-token\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.620188 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.620164    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr6tk\" (UniqueName: \"kubernetes.io/projected/92e6fabb-7410-46bc-a67f-40d9116f02eb-kube-api-access-zr6tk\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.687967 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.672112    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:13.694381 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.694335    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:46:13.694523 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.694425    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6" podUID="68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:46:13.865989 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:13.865956    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:14.031406 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:14.011250    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"]
Nov 03 19:46:14.048921 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:14.048879    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:46:14.049146 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:14.049129    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:46:14.049267 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:14.049253    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:46:14.653159 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:14.653117    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq" event={"ID":"92e6fabb-7410-46bc-a67f-40d9116f02eb","Type":"ContainerStarted","Data":"ac47927c63e565e280a9d248a2bf479df86334a23d0cf5b7b0b63b62fd0e4378"}
Nov 03 19:46:14.653550 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:14.653184    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq" event={"ID":"92e6fabb-7410-46bc-a67f-40d9116f02eb","Type":"ContainerStarted","Data":"92e1385a647a15006c394d92d2bba135fd72e1f86881c23e9e25efed5959a1b0"}
Nov 03 19:46:14.683914 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:14.678964    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:14.696010 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:14.695975    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:46:14.696187 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:14.696033    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6" podUID="68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:46:14.991910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:14.991051    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:15.557082 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:15.557049    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:15.682859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:15.677497    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:46:15.682859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:15.677566    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq" podUID="92e6fabb-7410-46bc-a67f-40d9116f02eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 19:46:15.722862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:15.714887    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:46:15.751857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:15.744239    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:46:15.766325 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:15.766256    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq" podStartSLOduration=2.766235762 podStartE2EDuration="2.766235762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:14.691027602 +0000 UTC m=+1130.945334828" watchObservedRunningTime="2026-11-03 19:46:15.766235762 +0000 UTC m=+1132.020542991"
Nov 03 19:46:16.162785 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:16.162748    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:16.458298 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:16.458205    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:16.681860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:16.674281    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:46:16.681860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:16.674364    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq" podUID="92e6fabb-7410-46bc-a67f-40d9116f02eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 19:46:17.140961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:17.140926    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:17.692862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:17.687248    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:46:17.692862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:17.687328    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq" podUID="92e6fabb-7410-46bc-a67f-40d9116f02eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 19:46:17.796507 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:17.796425    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:18.585757 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:18.585719    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:18.676260 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:18.676219    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:46:18.676589 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:18.676547    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq" podUID="92e6fabb-7410-46bc-a67f-40d9116f02eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 19:46:19.689283 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:19.689241    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:46:19.690080 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:19.690041    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq" podUID="92e6fabb-7410-46bc-a67f-40d9116f02eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 19:46:19.751248 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:19.751212    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:20.514505 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:20.514468    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:20.663730 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:20.663699    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:20.681859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:20.674611    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:46:20.681859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:20.674675    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq" podUID="92e6fabb-7410-46bc-a67f-40d9116f02eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 19:46:21.376322 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:21.376279    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:21.698370 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:21.698260    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:21.702486 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:21.702464    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:21.710420 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:21.710384    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:46:22.241148 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:22.241044    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:23.027397 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:23.027354    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:23.223219 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:23.223183    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:23.866579 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:23.866546    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:24.244254 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.244163    5237 scope.go:122] "RemoveContainer" containerID="204031757d4e5cb5ae27aa4195eab67c965b77a8d59466746262f52110259537"
Nov 03 19:46:24.474992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.473735    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"]
Nov 03 19:46:24.482488 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.482459    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.486074 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.486039    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"]
Nov 03 19:46:24.493864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.493821    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-eae28ff0\"/\"kube-root-ca.crt\""
Nov 03 19:46:24.494243 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.494223    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-eae28ff0\"/\"default-dockercfg-gzffp\""
Nov 03 19:46:24.494495 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.494446    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-eae28ff0\"/\"e2e-lifecycle-model-eae28ff0-kserve-self-signed-certs\""
Nov 03 19:46:24.494656 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.494641    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-eae28ff0\"/\"openshift-service-ca.crt\""
Nov 03 19:46:24.501160 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.501131    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:24.587042 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.586997    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-home\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.587257 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.587231    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-tls-certs\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.587408 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.587389    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-tmp-dir\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.587462 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.587447    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-dshm\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.587594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.587570    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-model-cache\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.587713 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.587691    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghxt6\" (UniqueName: \"kubernetes.io/projected/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-kube-api-access-ghxt6\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.689989 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.689362    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-tmp-dir\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.689989 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.689438    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-dshm\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.689989 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.689471    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-model-cache\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.689989 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.689548    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghxt6\" (UniqueName: \"kubernetes.io/projected/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-kube-api-access-ghxt6\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.689989 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.689660    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-home\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.689989 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.689785    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-tls-certs\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.694863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.690739    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-model-cache\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.694863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.691169    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-tmp-dir\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.717864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.696335    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-dshm\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.717864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.703051    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-home\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.717864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.708344    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-tls-certs\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.717864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.708998    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghxt6\" (UniqueName: \"kubernetes.io/projected/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-kube-api-access-ghxt6\") pod \"e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") " pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:24.817876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:24.817174    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:25.372994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:25.372502    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:25.632487 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:25.632453    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"]
Nov 03 19:46:25.640477 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:46:25.640448    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod568c2367_6f5c_423f_9c5b_5cc760e4cb7b.slice/crio-f9603ac8f443daf7e61f8b6088988afe84e968a8a2e07d49d324b6a78543bd17 WatchSource:0}: Error finding container f9603ac8f443daf7e61f8b6088988afe84e968a8a2e07d49d324b6a78543bd17: Status 404 returned error can't find the container with id f9603ac8f443daf7e61f8b6088988afe84e968a8a2e07d49d324b6a78543bd17
Nov 03 19:46:25.986885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:25.986754    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6" event={"ID":"568c2367-6f5c-423f-9c5b-5cc760e4cb7b","Type":"ContainerStarted","Data":"9b637d2afd0d34cd5e58421a8d27b7cbb008af86fcc9cc7fdce82f17afa3eb93"}
Nov 03 19:46:25.986885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:25.986811    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6" event={"ID":"568c2367-6f5c-423f-9c5b-5cc760e4cb7b","Type":"ContainerStarted","Data":"f9603ac8f443daf7e61f8b6088988afe84e968a8a2e07d49d324b6a78543bd17"}
Nov 03 19:46:26.012702 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:26.011897    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6" podStartSLOduration=2.011878763 podStartE2EDuration="2.011878763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:26.00720626 +0000 UTC m=+1142.261513487" watchObservedRunningTime="2026-11-03 19:46:26.011878763 +0000 UTC m=+1142.266185988"
Nov 03 19:46:26.301882 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:26.301438    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:26.827860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:26.821947    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:27.380520 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:27.380487    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:28.556115 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:28.556080    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:29.258545 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:29.254249    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:29.905058 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:29.905011    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:30.327261 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:30.327223    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:30.638498 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:30.638408    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:31.526553 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:31.526509    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:32.274642 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:32.274605    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:32.939940 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:32.939902    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:33.614438 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:33.614405    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:34.338173 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:34.338063    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:34.818080 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:34.818036    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:34.832032 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:34.832003    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:35.099442 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:35.099350    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:35.208450 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:35.208420    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:46:35.858239 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:35.858192    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:36.537659 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:36.537617    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:37.331596 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:37.331557    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:37.985415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:37.985374    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:38.785268 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:38.785229    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:39.399063 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:39.399026    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:40.034297 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:40.034265    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:40.722861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:40.721589    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:41.421028 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:41.420986    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:42.081976 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:42.081938    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:42.659650 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:42.659619    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:43.260689 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:46:43.260646    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:46:43.424871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:43.420827    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:44.144383 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:44.144351    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:44.830621 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:44.830584    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:45.519416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:45.519376    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:46.125661 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:46.125624    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:46.713310 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:46.713272    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:47.342819 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:47.342784    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:47.949992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:47.949948    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:48.650084 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:48.650047    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:49.411788 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:49.411748    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:50.114347 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:50.114307    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:50.738625 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:50.738587    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:51.511643 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:51.511607    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:52.119985 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:52.119936    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:52.717448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:52.717406    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:53.336336 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:53.336302    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:54.112515 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:54.112466    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:54.729451 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:54.729417    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:55.346072 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:55.346034    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:56.061767 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:56.061670    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:56.683568 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:56.683528    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:57.591335 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:57.591289    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:58.501563 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.501530    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"]
Nov 03 19:46:58.506801 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.506773    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.532510 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.532476    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"]
Nov 03 19:46:58.558649 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.558610    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:58.683859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.623975    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-podinfo\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.684550 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.684517    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-workload-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.689066 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.689032    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fdb8431-30cd-4023-94e9-47f32412e2e7-istiod-ca-cert\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.697865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.692941    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-data\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.698135 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.698109    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-token\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.708997 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.708964    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-workload-certs\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.709332 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.709309    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-credential-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.709617 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.709598    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-envoy\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.719083 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.719053    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnpzf\" (UniqueName: \"kubernetes.io/projected/5fdb8431-30cd-4023-94e9-47f32412e2e7-kube-api-access-vnpzf\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.850210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.836045    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-envoy\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.850635 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.850598    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnpzf\" (UniqueName: \"kubernetes.io/projected/5fdb8431-30cd-4023-94e9-47f32412e2e7-kube-api-access-vnpzf\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.851023 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.850999    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-podinfo\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.851266 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.851246    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-workload-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.854809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.854783    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fdb8431-30cd-4023-94e9-47f32412e2e7-istiod-ca-cert\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.856008 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.854640    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-workload-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.856166 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.855899    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fdb8431-30cd-4023-94e9-47f32412e2e7-istiod-ca-cert\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.856656 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.856634    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-data\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.857097 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.857078    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-token\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.857586 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.857555    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-workload-certs\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.858102 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.858084    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-credential-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.858721 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.857995    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-workload-certs\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.858872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.857019    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-data\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.859005 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.858476    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-credential-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.889779 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.889724    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-podinfo\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.891793 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.891711    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnpzf\" (UniqueName: \"kubernetes.io/projected/5fdb8431-30cd-4023-94e9-47f32412e2e7-kube-api-access-vnpzf\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.936943 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.936821    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-envoy\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:58.942530 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:58.942498    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-token\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:59.120773 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:59.120675    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:46:59.711691 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:59.711641    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:46:59.712137 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:59.711743    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:46:59.712137 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:59.711792    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:46:59.714298 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:59.714265    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"]
Nov 03 19:46:59.774777 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:46:59.774530    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:00.633861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:00.632853    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:00.819946 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:00.811965    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb" event={"ID":"5fdb8431-30cd-4023-94e9-47f32412e2e7","Type":"ContainerStarted","Data":"97e20ca505dac2ef06e549c4f26275d9a4ee6c1b6a38b14f35c32e193459ded1"}
Nov 03 19:47:00.819946 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:00.812119    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb" event={"ID":"5fdb8431-30cd-4023-94e9-47f32412e2e7","Type":"ContainerStarted","Data":"c2a2f11f08067eb422df67de64971d4c36124743417f72fd91db7fa7d79a716f"}
Nov 03 19:47:01.134882 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:01.122901    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:47:01.134882 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:01.123731    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:47:01.134882 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:01.123859    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb" podUID="5fdb8431-30cd-4023-94e9-47f32412e2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:47:01.942870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:01.939392    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:01.989860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:01.984116    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:01.992243 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:01.992203    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:47:02.127718 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:02.127676    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:47:02.128030 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:02.127999    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb" podUID="5fdb8431-30cd-4023-94e9-47f32412e2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:47:02.128534 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:02.128508    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 19:47:02.181076 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:02.181007    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:02.219123 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:02.218991    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6" podUID="568c2367-6f5c-423f-9c5b-5cc760e4cb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:02.246010 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:02.245959    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:02.248328 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:02.248285    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:02.248939 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:02.248906    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 19:47:02.250963 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:02.250349    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:02.323860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:02.309768    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:02.323860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:02.318923    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:02.337210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:02.337164    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:02.639520 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:02.639471    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:02.949924 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:02.880946    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:03.042912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:02.992929    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:03.042912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:02.993514    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:47:03.042912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.002418    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:03.042912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.004309    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:47:03.209631 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.209536    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:47:03.210360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.210336    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:47:03.238974 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.238917    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6" podUID="568c2367-6f5c-423f-9c5b-5cc760e4cb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:03.240216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.240194    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:47:03.242184 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.241254    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:03.242184 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.241703    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:47:03.247228 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.246902    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:03.247359 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.247241    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:47:03.270869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.253552    5237 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded"
Nov 03 19:47:03.270869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.253759    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:03.270869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.254076    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:47:03.270869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.254483    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:03.271768 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.271367    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:47:03.287948 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.287922    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:47:03.334103 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.330977    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:03.342209 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.342147    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb" podStartSLOduration=5.342126509 podStartE2EDuration="5.342126509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:00.870111747 +0000 UTC m=+1177.124418994" watchObservedRunningTime="2026-11-03 19:47:03.342126509 +0000 UTC m=+1179.596433735"
Nov 03 19:47:03.361136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.353607    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:47:03.361136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.342765    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:47:03.361136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.354026    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb" podUID="5fdb8431-30cd-4023-94e9-47f32412e2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:47:03.361136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.342493    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:03.361136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.360736    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:47:03.433369 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.433339    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:47:03.433681 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.433647    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:47:03.434353 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.434222    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:47:03.455790 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.437461    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:47:03.455790 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.454606    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:47:03.455790 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.454663    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:47:03.455790 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.454698    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:47:03.455790 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.454730    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:47:03.478919 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.463338    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:47:03.563125 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:03.561004    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:04.121664 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:04.121622    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:47:04.122240 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:04.121676    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb" podUID="5fdb8431-30cd-4023-94e9-47f32412e2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:47:04.882917 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:04.878311    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:05.121246 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:05.121212    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:47:05.121427 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:05.121266    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb" podUID="5fdb8431-30cd-4023-94e9-47f32412e2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:47:05.787861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:05.786500    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:06.121912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:06.121781    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:47:06.121912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:06.121872    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb" podUID="5fdb8431-30cd-4023-94e9-47f32412e2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:47:06.252803 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:06.252763    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:06.713986 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:06.713947    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:06.886638 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:06.886601    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:07.340608 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:07.340576    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:47:07.343059 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:07.342230    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:47:07.344354 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:07.344327    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:47:07.765663 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:07.764237    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:08.630813 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:08.630728    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:09.498150 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:09.498029    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:10.428867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:10.418569    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:11.150705 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:11.150659    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:11.362101 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:11.362050    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:12.209857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.202398    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:12.473784 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.473691    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:12.524136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.524096    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"]
Nov 03 19:47:12.528530 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.528503    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"
Nov 03 19:47:12.536465 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.536418    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f292e03c\""
Nov 03 19:47:12.537787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.537762    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f292e03c-dockercfg-w75wj\""
Nov 03 19:47:12.542533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.542510    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"]
Nov 03 19:47:12.566583 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.566547    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"]
Nov 03 19:47:12.571856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.571822    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"
Nov 03 19:47:12.579407 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.579375    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"]
Nov 03 19:47:12.595573 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.595540    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80c1008-2bd2-4a10-bacf-ec1d7148ebf8-plugins-config-volume\") pod \"payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp\" (UID: \"e80c1008-2bd2-4a10-bacf-ec1d7148ebf8\") " pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"
Nov 03 19:47:12.596051 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.596026    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhzc6\" (UniqueName: \"kubernetes.io/projected/7f80a141-ae01-4ca9-911f-f2fe09d5492b-kube-api-access-xhzc6\") pod \"payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk\" (UID: \"7f80a141-ae01-4ca9-911f-f2fe09d5492b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"
Nov 03 19:47:12.596291 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.596263    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjnrn\" (UniqueName: \"kubernetes.io/projected/e80c1008-2bd2-4a10-bacf-ec1d7148ebf8-kube-api-access-jjnrn\") pod \"payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp\" (UID: \"e80c1008-2bd2-4a10-bacf-ec1d7148ebf8\") " pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"
Nov 03 19:47:12.596543 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.596524    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f80a141-ae01-4ca9-911f-f2fe09d5492b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk\" (UID: \"7f80a141-ae01-4ca9-911f-f2fe09d5492b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"
Nov 03 19:47:12.713856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.713656    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:12.759910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.755116    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f80a141-ae01-4ca9-911f-f2fe09d5492b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk\" (UID: \"7f80a141-ae01-4ca9-911f-f2fe09d5492b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"
Nov 03 19:47:12.759910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.755285    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80c1008-2bd2-4a10-bacf-ec1d7148ebf8-plugins-config-volume\") pod \"payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp\" (UID: \"e80c1008-2bd2-4a10-bacf-ec1d7148ebf8\") " pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"
Nov 03 19:47:12.759910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.755518    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhzc6\" (UniqueName: \"kubernetes.io/projected/7f80a141-ae01-4ca9-911f-f2fe09d5492b-kube-api-access-xhzc6\") pod \"payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk\" (UID: \"7f80a141-ae01-4ca9-911f-f2fe09d5492b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"
Nov 03 19:47:12.759910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.755576    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjnrn\" (UniqueName: \"kubernetes.io/projected/e80c1008-2bd2-4a10-bacf-ec1d7148ebf8-kube-api-access-jjnrn\") pod \"payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp\" (UID: \"e80c1008-2bd2-4a10-bacf-ec1d7148ebf8\") " pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"
Nov 03 19:47:12.759910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.756823    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f80a141-ae01-4ca9-911f-f2fe09d5492b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk\" (UID: \"7f80a141-ae01-4ca9-911f-f2fe09d5492b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"
Nov 03 19:47:12.759910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.757615    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80c1008-2bd2-4a10-bacf-ec1d7148ebf8-plugins-config-volume\") pod \"payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp\" (UID: \"e80c1008-2bd2-4a10-bacf-ec1d7148ebf8\") " pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"
Nov 03 19:47:12.917589 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.917541    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjnrn\" (UniqueName: \"kubernetes.io/projected/e80c1008-2bd2-4a10-bacf-ec1d7148ebf8-kube-api-access-jjnrn\") pod \"payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp\" (UID: \"e80c1008-2bd2-4a10-bacf-ec1d7148ebf8\") " pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"
Nov 03 19:47:12.925137 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:12.922587    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhzc6\" (UniqueName: \"kubernetes.io/projected/7f80a141-ae01-4ca9-911f-f2fe09d5492b-kube-api-access-xhzc6\") pod \"payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk\" (UID: \"7f80a141-ae01-4ca9-911f-f2fe09d5492b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"
Nov 03 19:47:13.145443 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:13.145406    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"
Nov 03 19:47:13.190061 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:13.190023    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"
Nov 03 19:47:13.268479 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:47:13.268442    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:47:13.434886 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:47:13.434474    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f80a141_ae01_4ca9_911f_f2fe09d5492b.slice/crio-8045e27005b6e00d936e646c0ca7647dfa6cd08358ac48c86c634ddcbb25e91f WatchSource:0}: Error finding container 8045e27005b6e00d936e646c0ca7647dfa6cd08358ac48c86c634ddcbb25e91f: Status 404 returned error can't find the container with id 8045e27005b6e00d936e646c0ca7647dfa6cd08358ac48c86c634ddcbb25e91f
Nov 03 19:47:13.450213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:13.443985    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"]
Nov 03 19:47:13.529974 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:13.466982    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:13.598264 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:13.598224    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"]
Nov 03 19:47:14.175866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:14.169048    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:14.195870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:14.194424    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp" event={"ID":"e80c1008-2bd2-4a10-bacf-ec1d7148ebf8","Type":"ContainerStarted","Data":"2e406443618c8fb1896643c1b0aab402a0a83c70ea059d128c8734d02fbaab42"}
Nov 03 19:47:14.195870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:14.194506    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp" event={"ID":"e80c1008-2bd2-4a10-bacf-ec1d7148ebf8","Type":"ContainerStarted","Data":"0e4a6cc4452195b020e5875ec70bef5cf69bec838c459887f8d4dfa1fd38c610"}
Nov 03 19:47:14.200552 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:14.196747    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"
Nov 03 19:47:14.215860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:14.215696    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk" event={"ID":"7f80a141-ae01-4ca9-911f-f2fe09d5492b","Type":"ContainerStarted","Data":"40e9571065bb0d9e5f721144659703671bb70f494e2172efdf9eaa04e75c66cb"}
Nov 03 19:47:14.215860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:14.215768    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk" event={"ID":"7f80a141-ae01-4ca9-911f-f2fe09d5492b","Type":"ContainerStarted","Data":"8045e27005b6e00d936e646c0ca7647dfa6cd08358ac48c86c634ddcbb25e91f"}
Nov 03 19:47:14.223621 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:14.216266    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"
Nov 03 19:47:14.240359 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:14.236467    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp" podStartSLOduration=2.236449253 podStartE2EDuration="2.236449253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:14.235586761 +0000 UTC m=+1190.489893986" watchObservedRunningTime="2026-11-03 19:47:14.236449253 +0000 UTC m=+1190.490756481"
Nov 03 19:47:14.270214 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:14.270160    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk" podStartSLOduration=2.270145201 podStartE2EDuration="2.270145201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:14.266189738 +0000 UTC m=+1190.520496966" watchObservedRunningTime="2026-11-03 19:47:14.270145201 +0000 UTC m=+1190.524452425"
Nov 03 19:47:14.964938 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:14.964432    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:15.714527 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:15.714498    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:16.416783 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:16.404036    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:17.153926 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:17.151725    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:17.829474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:17.829438    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:18.511022 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:18.510984    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:19.236730 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:19.236690    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:19.842557 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:19.842495    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:20.532741 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:20.532704    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:20.668621 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:20.668582    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:21.547072 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:21.547040    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:22.318569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:22.318535    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:22.948847 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:22.948796    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:23.646505 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:23.646471    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:24.281586 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:24.281548    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:24.944390 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:24.944339    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:25.226404 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:25.226325    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"
Nov 03 19:47:25.673859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:25.670460    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:26.234055 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:26.234022    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"
Nov 03 19:47:26.327004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:26.326965    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:26.929414 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:26.929369    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:27.526105 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:27.526074    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:28.136054 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:28.136020    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:29.011526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:29.011488    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:29.716856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:29.711786    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:30.355870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:30.354410    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:30.499775 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:30.499738    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:31.262946 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:31.262899    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:32.043269 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:32.043235    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:32.832483 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:32.832443    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:33.526791 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:33.526753    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:34.223571 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:34.223537    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:34.828546 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:34.828506    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:35.491072 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:35.491038    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:36.538310 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:36.538273    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:36.682110 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:36.682073    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:37.428495 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:37.428416    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:37.647744 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:37.647707    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:38.183307 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:38.183259    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:38.428045 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:38.428006    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:39.214362 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:39.214324    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:39.916096 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:39.916056    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:40.710344 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:40.709710    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:41.375126 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:41.375071    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:42.154697 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:42.154634    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:42.789453 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:42.789415    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:43.259983 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:47:43.259898    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:47:43.416732 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:43.416696    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:44.146556 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:44.146514    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:44.788567 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:44.788529    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:44.965852 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:44.965803    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:45.232824 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:45.232721    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:45.523249 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:45.523165    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:46.258058 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:46.258020    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:47.037567 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:47.037535    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:47.815661 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:47.815620    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:48.598155 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:48.598114    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:49.210448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:49.210411    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:49.477540 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:49.477464    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7095-165890"]
Nov 03 19:47:49.483624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:49.483595    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-165890"
Nov 03 19:47:49.484606 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:49.484579    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-165890"]
Nov 03 19:47:49.487515 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:49.487497    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hjqcx\""
Nov 03 19:47:49.606135 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:49.606096    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5g58\" (UniqueName: \"kubernetes.io/projected/ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0-kube-api-access-d5g58\") pod \"test-curl-7095-165890\" (UID: \"ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0\") " pod="opendatahub/test-curl-7095-165890"
Nov 03 19:47:49.707625 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:49.707585    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5g58\" (UniqueName: \"kubernetes.io/projected/ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0-kube-api-access-d5g58\") pod \"test-curl-7095-165890\" (UID: \"ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0\") " pod="opendatahub/test-curl-7095-165890"
Nov 03 19:47:49.720142 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:49.720112    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5g58\" (UniqueName: \"kubernetes.io/projected/ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0-kube-api-access-d5g58\") pod \"test-curl-7095-165890\" (UID: \"ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0\") " pod="opendatahub/test-curl-7095-165890"
Nov 03 19:47:49.799231 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:49.799204    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-165890"
Nov 03 19:47:49.909882 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:49.906257    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:50.222810 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:50.222780    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-165890"]
Nov 03 19:47:50.225406 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:47:50.225378    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded0b80ca_3ec6_4f4a_82f7_23c03f5096b0.slice/crio-3a7875f4d1c9e6a34409eb4a415d8441c615d7aabe71971938804023c66dd54f WatchSource:0}: Error finding container 3a7875f4d1c9e6a34409eb4a415d8441c615d7aabe71971938804023c66dd54f: Status 404 returned error can't find the container with id 3a7875f4d1c9e6a34409eb4a415d8441c615d7aabe71971938804023c66dd54f
Nov 03 19:47:50.448539 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:50.448508    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:51.014912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:51.014872    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-165890" event={"ID":"ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0","Type":"ContainerStarted","Data":"ec607e6fb39c95f452bd4300b09eb63d64fdf8956295a2033378f128fb5a4ea2"}
Nov 03 19:47:51.015095 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:51.014926    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-165890" event={"ID":"ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0","Type":"ContainerStarted","Data":"3a7875f4d1c9e6a34409eb4a415d8441c615d7aabe71971938804023c66dd54f"}
Nov 03 19:47:51.035850 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:51.035786    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7095-165890" podStartSLOduration=2.035772583 podStartE2EDuration="2.035772583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:51.031771301 +0000 UTC m=+1227.286078538" watchObservedRunningTime="2026-11-03 19:47:51.035772583 +0000 UTC m=+1227.290079808"
Nov 03 19:47:51.226052 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:51.225996    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:51.982986 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:51.982953    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7095-165890"]
Nov 03 19:47:51.986904 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:51.986874    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7095-165890"]
Nov 03 19:47:52.232862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:52.232804    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:52.606811 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:52.606772    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:53.027736 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:53.027647    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7095-165890" podUID="ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0" containerName="test-curl-7095-165890" containerID="cri-o://ec607e6fb39c95f452bd4300b09eb63d64fdf8956295a2033378f128fb5a4ea2" gracePeriod=2
Nov 03 19:47:53.030733 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:47:53.030704    5237 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7095-165890\" is forbidden: User \"system:node:ip-10-0-133-29.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-29.us-east-2.compute.internal' and this object" podUID="ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0" pod="opendatahub/test-curl-7095-165890"
Nov 03 19:47:53.319622 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:53.319586    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:53.860434 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:47:53.860393    5237 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7095-165890\" is forbidden: User \"system:node:ip-10-0-133-29.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-29.us-east-2.compute.internal' and this object" podUID="ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0" pod="opendatahub/test-curl-7095-165890"
Nov 03 19:47:53.926676 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:53.926639    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:54.603334 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:54.603294    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:55.252408 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:55.252376    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-165890"
Nov 03 19:47:55.340401 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:55.340361    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:55.376858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:55.365635    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0-kube-api-access-d5g58\" (UniqueName: \"kubernetes.io/projected/ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0-kube-api-access-d5g58\") pod \"ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0\" (UID: \"ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0\") "
Nov 03 19:47:55.400692 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:55.385044    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0-kube-api-access-d5g58" pod "ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0" (UID: "ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0"). InnerVolumeSpecName "kube-api-access-d5g58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:55.472807 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:55.471426    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5g58\" (UniqueName: \"kubernetes.io/projected/ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0-kube-api-access-d5g58\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:55.553439 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:55.553404    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:55.862985 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:55.862952    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0" path="/var/lib/kubelet/pods/ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0/volumes"
Nov 03 19:47:56.075845 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:56.075810    5237 generic.go:356] "Generic (PLEG): container finished" podID="ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0" containerID="ec607e6fb39c95f452bd4300b09eb63d64fdf8956295a2033378f128fb5a4ea2" exitCode=137
Nov 03 19:47:56.076062 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:56.075887    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-165890"
Nov 03 19:47:56.076062 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:56.075905    5237 scope.go:122] "RemoveContainer" containerID="ec607e6fb39c95f452bd4300b09eb63d64fdf8956295a2033378f128fb5a4ea2"
Nov 03 19:47:56.129922 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:56.114508    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:56.219054 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:56.212391    5237 scope.go:122] "RemoveContainer" containerID="ec607e6fb39c95f452bd4300b09eb63d64fdf8956295a2033378f128fb5a4ea2"
Nov 03 19:47:56.219054 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:47:56.213029    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec607e6fb39c95f452bd4300b09eb63d64fdf8956295a2033378f128fb5a4ea2\": container with ID starting with ec607e6fb39c95f452bd4300b09eb63d64fdf8956295a2033378f128fb5a4ea2 not found: ID does not exist" containerID="ec607e6fb39c95f452bd4300b09eb63d64fdf8956295a2033378f128fb5a4ea2"
Nov 03 19:47:56.219054 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:56.213067    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec607e6fb39c95f452bd4300b09eb63d64fdf8956295a2033378f128fb5a4ea2"} err="failed to get container status \"ec607e6fb39c95f452bd4300b09eb63d64fdf8956295a2033378f128fb5a4ea2\": rpc error: code = NotFound desc = could not find container \"ec607e6fb39c95f452bd4300b09eb63d64fdf8956295a2033378f128fb5a4ea2\": container with ID starting with ec607e6fb39c95f452bd4300b09eb63d64fdf8956295a2033378f128fb5a4ea2 not found: ID does not exist"
Nov 03 19:47:56.255859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:56.255408    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:56.499280 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:56.482554    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:57.082994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:57.082955    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:57.336975 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:57.330459    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:57.961861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:57.961409    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:58.826595 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:58.826554    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:59.604796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:59.604752    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:59.758866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:47:59.747572    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:00.833124 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:00.833079    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:00.998388 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:00.991322    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:01.019044 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.019010    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"]
Nov 03 19:48:01.020072 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.020046    5237 cpu_manager.go:414] "containerMap: removing container" podUID="ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0" containerName="test-curl-7095-165890"
Nov 03 19:48:01.020072 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.020072    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0" containerName="test-curl-7095-165890"
Nov 03 19:48:01.020272 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.020258    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed0b80ca-3ec6-4f4a-82f7-23c03f5096b0" containerName="test-curl-7095-165890"
Nov 03 19:48:01.025996 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.025973    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"
Nov 03 19:48:01.034144 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.034118    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-29ab98\""
Nov 03 19:48:01.034303 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.034252    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-29ab98-dockercfg-vh2jv\""
Nov 03 19:48:01.037175 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.037135    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"]
Nov 03 19:48:01.165921 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.162456    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:01.198859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.183349    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdbdm\" (UniqueName: \"kubernetes.io/projected/adf802a7-4ee5-439e-8748-8d390c5f1f51-kube-api-access-pdbdm\") pod \"payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97\" (UID: \"adf802a7-4ee5-439e-8748-8d390c5f1f51\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"
Nov 03 19:48:01.198859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.183506    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf802a7-4ee5-439e-8748-8d390c5f1f51-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97\" (UID: \"adf802a7-4ee5-439e-8748-8d390c5f1f51\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"
Nov 03 19:48:01.290860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.289745    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdbdm\" (UniqueName: \"kubernetes.io/projected/adf802a7-4ee5-439e-8748-8d390c5f1f51-kube-api-access-pdbdm\") pod \"payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97\" (UID: \"adf802a7-4ee5-439e-8748-8d390c5f1f51\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"
Nov 03 19:48:01.290860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.289950    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf802a7-4ee5-439e-8748-8d390c5f1f51-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97\" (UID: \"adf802a7-4ee5-439e-8748-8d390c5f1f51\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"
Nov 03 19:48:01.294545 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.291204    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf802a7-4ee5-439e-8748-8d390c5f1f51-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97\" (UID: \"adf802a7-4ee5-439e-8748-8d390c5f1f51\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"
Nov 03 19:48:01.340871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.337908    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdbdm\" (UniqueName: \"kubernetes.io/projected/adf802a7-4ee5-439e-8748-8d390c5f1f51-kube-api-access-pdbdm\") pod \"payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97\" (UID: \"adf802a7-4ee5-439e-8748-8d390c5f1f51\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"
Nov 03 19:48:01.350688 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.349522    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"
Nov 03 19:48:01.798820 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.798785    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"]
Nov 03 19:48:01.799650 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:48:01.799620    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf802a7_4ee5_439e_8748_8d390c5f1f51.slice/crio-f06f1fbc939f17210784424f271d642701c6902486f3413bade7eb55d9458e23 WatchSource:0}: Error finding container f06f1fbc939f17210784424f271d642701c6902486f3413bade7eb55d9458e23: Status 404 returned error can't find the container with id f06f1fbc939f17210784424f271d642701c6902486f3413bade7eb55d9458e23
Nov 03 19:48:01.914514 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:01.914472    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:02.263121 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:02.263038    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97" event={"ID":"adf802a7-4ee5-439e-8748-8d390c5f1f51","Type":"ContainerStarted","Data":"74eb9d2e40a7eb0bcbc1c769d1caffa88a23fb3f8f2e4816090955bca0c502c3"}
Nov 03 19:48:02.263121 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:02.263086    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97" event={"ID":"adf802a7-4ee5-439e-8748-8d390c5f1f51","Type":"ContainerStarted","Data":"f06f1fbc939f17210784424f271d642701c6902486f3413bade7eb55d9458e23"}
Nov 03 19:48:02.263318 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:02.263172    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"
Nov 03 19:48:02.286024 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:02.285971    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97" podStartSLOduration=2.285954566 podStartE2EDuration="2.285954566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:02.282428846 +0000 UTC m=+1238.536736073" watchObservedRunningTime="2026-11-03 19:48:02.285954566 +0000 UTC m=+1238.540261832"
Nov 03 19:48:02.815358 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:02.815320    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:03.033516 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:03.033475    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:03.163256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:03.163150    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:03.629395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:03.629363    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:03.961357 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:03.961269    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:04.656420 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:04.656379    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:05.407799 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:05.407763    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:06.051035 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:06.050999    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:06.651892 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:06.651829    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:07.438944 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:07.438909    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:08.112962 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:08.112924    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:08.718431 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:08.718380    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:09.302594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:09.302557    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:09.437771 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:09.437737    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:09.571650 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:09.571536    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:10.067072 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:10.066812    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:11.023455 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:11.023423    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:11.778546 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:11.778458    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:12.438175 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:12.438132    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:13.029698 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:13.029664    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:13.257275 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:48:13.257236    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:48:13.274380 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:13.274355    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"
Nov 03 19:48:13.657927 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:13.657848    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:14.339963 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:14.339922    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:15.033400 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:15.033365    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:15.210130 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:15.210097    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:15.708538 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:15.708479    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:16.324931 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:16.324895    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:17.430367 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:17.430329    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:18.351590 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:18.351555    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:19.147922 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:19.147869    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:19.893296 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:19.893258    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:21.045857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:21.045472    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:21.579706 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:21.579670    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:21.771556 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:21.771517    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:22.043908 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:22.042510    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:22.326203 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:22.326113    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:23.227489 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:23.227450    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:24.038039 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:24.037998    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:24.811797 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:24.811752    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:24.954868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:24.954816    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:25.443408 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:25.443371    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:25.583391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:25.583354    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:25.979859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:25.974320    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:26.453410 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:26.453373    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:26.600374 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:26.600340    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:27.337745 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:27.337711    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:28.213882 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:28.213811    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:29.159074 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:29.159039    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:29.449825 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:29.449740    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:29.640036 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:29.640002    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:30.018784 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:30.018752    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:30.170171 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:30.170133    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:30.328069 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:30.328029    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:30.496175 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:30.496140    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:31.338326 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:31.338271    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:31.502995 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:31.502948    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:31.529049 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:31.526124    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"]
Nov 03 19:48:31.533734 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:31.533709    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"
Nov 03 19:48:31.549140 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:31.549112    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-25944c-dockercfg-qpvk4\""
Nov 03 19:48:31.549299 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:31.549260    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-25944c\""
Nov 03 19:48:31.557970 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:31.557942    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"]
Nov 03 19:48:31.576117 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:31.576088    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"]
Nov 03 19:48:31.576117 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:31.576127    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"]
Nov 03 19:48:31.576363 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:31.576207    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"
Nov 03 19:48:31.927967 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:31.927315    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcgzg\" (UniqueName: \"kubernetes.io/projected/f4d213de-c3e2-4455-ab48-e78d0f72b169-kube-api-access-tcgzg\") pod \"payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv\" (UID: \"f4d213de-c3e2-4455-ab48-e78d0f72b169\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"
Nov 03 19:48:31.927967 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:31.927385    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d213de-c3e2-4455-ab48-e78d0f72b169-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv\" (UID: \"f4d213de-c3e2-4455-ab48-e78d0f72b169\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"
Nov 03 19:48:31.927967 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:31.927764    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b14de1-629b-4f1a-b35c-cd23b30eff51-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b\" (UID: \"f6b14de1-629b-4f1a-b35c-cd23b30eff51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"
Nov 03 19:48:31.927967 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:31.927899    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mc9w\" (UniqueName: \"kubernetes.io/projected/f6b14de1-629b-4f1a-b35c-cd23b30eff51-kube-api-access-9mc9w\") pod \"payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b\" (UID: \"f6b14de1-629b-4f1a-b35c-cd23b30eff51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"
Nov 03 19:48:31.996569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:31.996531    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:32.029339 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.029304    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgzg\" (UniqueName: \"kubernetes.io/projected/f4d213de-c3e2-4455-ab48-e78d0f72b169-kube-api-access-tcgzg\") pod \"payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv\" (UID: \"f4d213de-c3e2-4455-ab48-e78d0f72b169\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"
Nov 03 19:48:32.029339 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.029351    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d213de-c3e2-4455-ab48-e78d0f72b169-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv\" (UID: \"f4d213de-c3e2-4455-ab48-e78d0f72b169\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"
Nov 03 19:48:32.029648 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.029567    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b14de1-629b-4f1a-b35c-cd23b30eff51-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b\" (UID: \"f6b14de1-629b-4f1a-b35c-cd23b30eff51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"
Nov 03 19:48:32.029648 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.029640    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mc9w\" (UniqueName: \"kubernetes.io/projected/f6b14de1-629b-4f1a-b35c-cd23b30eff51-kube-api-access-9mc9w\") pod \"payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b\" (UID: \"f6b14de1-629b-4f1a-b35c-cd23b30eff51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"
Nov 03 19:48:32.030324 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.030295    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d213de-c3e2-4455-ab48-e78d0f72b169-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv\" (UID: \"f4d213de-c3e2-4455-ab48-e78d0f72b169\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"
Nov 03 19:48:32.030498 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.030331    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b14de1-629b-4f1a-b35c-cd23b30eff51-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b\" (UID: \"f6b14de1-629b-4f1a-b35c-cd23b30eff51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"
Nov 03 19:48:32.041763 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.041737    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcgzg\" (UniqueName: \"kubernetes.io/projected/f4d213de-c3e2-4455-ab48-e78d0f72b169-kube-api-access-tcgzg\") pod \"payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv\" (UID: \"f4d213de-c3e2-4455-ab48-e78d0f72b169\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"
Nov 03 19:48:32.042673 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.042653    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mc9w\" (UniqueName: \"kubernetes.io/projected/f6b14de1-629b-4f1a-b35c-cd23b30eff51-kube-api-access-9mc9w\") pod \"payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b\" (UID: \"f6b14de1-629b-4f1a-b35c-cd23b30eff51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"
Nov 03 19:48:32.177829 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.177796    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"
Nov 03 19:48:32.191421 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.191397    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"
Nov 03 19:48:32.429114 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.429066    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:32.558873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.554265    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"]
Nov 03 19:48:32.581501 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.581471    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"]
Nov 03 19:48:32.593993 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:48:32.593958    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b14de1_629b_4f1a_b35c_cd23b30eff51.slice/crio-b9ee6b683d9e7a973e703602fe0382254a54fb9c603b48b926397b8a995bdf56 WatchSource:0}: Error finding container b9ee6b683d9e7a973e703602fe0382254a54fb9c603b48b926397b8a995bdf56: Status 404 returned error can't find the container with id b9ee6b683d9e7a973e703602fe0382254a54fb9c603b48b926397b8a995bdf56
Nov 03 19:48:32.892337 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.892245    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv" event={"ID":"f4d213de-c3e2-4455-ab48-e78d0f72b169","Type":"ContainerStarted","Data":"54668c64f608b0c67e643a38300b3dde1fed941ee5aa9670d918866418484835"}
Nov 03 19:48:32.892337 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.892299    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv" event={"ID":"f4d213de-c3e2-4455-ab48-e78d0f72b169","Type":"ContainerStarted","Data":"0f37bb2d858c6d75e08b1fd5fa9ec8d2374237cea7d8a66250aa1fdfe3f50eef"}
Nov 03 19:48:32.892557 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.892338    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"
Nov 03 19:48:32.894084 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.894060    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b" event={"ID":"f6b14de1-629b-4f1a-b35c-cd23b30eff51","Type":"ContainerStarted","Data":"685e181c6cd4f7242881b0c1979e1f07667f513e467c9567660b3be29d4d7d13"}
Nov 03 19:48:32.894208 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.894096    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b" event={"ID":"f6b14de1-629b-4f1a-b35c-cd23b30eff51","Type":"ContainerStarted","Data":"b9ee6b683d9e7a973e703602fe0382254a54fb9c603b48b926397b8a995bdf56"}
Nov 03 19:48:32.894208 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.894184    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"
Nov 03 19:48:32.912123 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.912068    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv" podStartSLOduration=1.91205512 podStartE2EDuration="1.91205512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:32.910957461 +0000 UTC m=+1269.165264686" watchObservedRunningTime="2026-11-03 19:48:32.91205512 +0000 UTC m=+1269.166362394"
Nov 03 19:48:32.933391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:32.933328    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b" podStartSLOduration=1.933313989 podStartE2EDuration="1.933313989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:32.929582824 +0000 UTC m=+1269.183890047" watchObservedRunningTime="2026-11-03 19:48:32.933313989 +0000 UTC m=+1269.187621214"
Nov 03 19:48:33.196749 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:33.196653    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:33.848971 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:33.848938    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:34.725989 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:34.725908    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:34.866197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:34.866160    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:35.832992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:35.832952    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:36.714510 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:36.714469    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:37.651920 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:37.651879    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:38.338512 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:38.338477    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:38.942057 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:38.942017    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:39.142040 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:39.141995    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:39.832466 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:39.832428    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:40.511793 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:40.511726    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:41.128467 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:41.128422    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:41.281453 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:41.281418    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:42.148855 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:42.141428    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:42.944367 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:42.944327    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:43.289778 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:48:43.289742    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:48:43.632626 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:43.632534    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:43.936137 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:43.936053    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"
Nov 03 19:48:43.936137 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:43.936121    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"
Nov 03 19:48:44.400241 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:44.400202    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:45.193860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:45.187645    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:45.816252 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:45.816213    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:46.445510 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:46.445450    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:47.148860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:47.143121    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:48.193384 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:48.193292    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:48.838990 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:48.838953    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:49.629883 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:49.628073    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:50.298747 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:50.298713    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:51.055905 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:51.053512    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:51.660325 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:51.660238    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:52.349047 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:52.349008    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:53.030917 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:53.030872    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:53.149227 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:53.149086    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7095-82ff58"]
Nov 03 19:48:53.155446 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:53.155401    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-82ff58"
Nov 03 19:48:53.160406 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:53.160358    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hjqcx\""
Nov 03 19:48:53.163575 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:53.163545    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-82ff58"]
Nov 03 19:48:53.274825 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:53.274793    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gl9r\" (UniqueName: \"kubernetes.io/projected/ed345c07-8bf9-4c64-b755-5b37e6198aba-kube-api-access-7gl9r\") pod \"test-curl-7095-82ff58\" (UID: \"ed345c07-8bf9-4c64-b755-5b37e6198aba\") " pod="opendatahub/test-curl-7095-82ff58"
Nov 03 19:48:53.377314 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:53.377209    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gl9r\" (UniqueName: \"kubernetes.io/projected/ed345c07-8bf9-4c64-b755-5b37e6198aba-kube-api-access-7gl9r\") pod \"test-curl-7095-82ff58\" (UID: \"ed345c07-8bf9-4c64-b755-5b37e6198aba\") " pod="opendatahub/test-curl-7095-82ff58"
Nov 03 19:48:53.424852 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:53.424798    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gl9r\" (UniqueName: \"kubernetes.io/projected/ed345c07-8bf9-4c64-b755-5b37e6198aba-kube-api-access-7gl9r\") pod \"test-curl-7095-82ff58\" (UID: \"ed345c07-8bf9-4c64-b755-5b37e6198aba\") " pod="opendatahub/test-curl-7095-82ff58"
Nov 03 19:48:53.477185 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:53.477140    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-82ff58"
Nov 03 19:48:53.886229 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:53.886204    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-82ff58"]
Nov 03 19:48:53.889401 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:48:53.889369    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded345c07_8bf9_4c64_b755_5b37e6198aba.slice/crio-51474c54c74a0b6f2f5f07c759ebdb7bfdbd06414b9e1e88698d8540be8b493b WatchSource:0}: Error finding container 51474c54c74a0b6f2f5f07c759ebdb7bfdbd06414b9e1e88698d8540be8b493b: Status 404 returned error can't find the container with id 51474c54c74a0b6f2f5f07c759ebdb7bfdbd06414b9e1e88698d8540be8b493b
Nov 03 19:48:54.172552 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:54.172506    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-82ff58" event={"ID":"ed345c07-8bf9-4c64-b755-5b37e6198aba","Type":"ContainerStarted","Data":"51474c54c74a0b6f2f5f07c759ebdb7bfdbd06414b9e1e88698d8540be8b493b"}
Nov 03 19:48:54.208668 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:54.208576    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:54.691403 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:54.691362    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:55.234859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:55.217197    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-82ff58" event={"ID":"ed345c07-8bf9-4c64-b755-5b37e6198aba","Type":"ContainerStarted","Data":"a25d35b26c6553d8d30042dba04568f3de77c74c8ecc0682993e0f3f689b39a5"}
Nov 03 19:48:55.412856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:55.403961    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7095-82ff58" podStartSLOduration=2.403941359 podStartE2EDuration="2.403941359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:55.265031193 +0000 UTC m=+1291.519338420" watchObservedRunningTime="2026-11-03 19:48:55.403941359 +0000 UTC m=+1291.658248585"
Nov 03 19:48:55.816494 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:55.816456    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:56.096330 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:56.096239    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:56.177159 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:56.177119    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7095-82ff58"]
Nov 03 19:48:56.186399 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:56.186369    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7095-82ff58"]
Nov 03 19:48:56.705586 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:56.704732    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:57.113521 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:57.113482    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:57.250692 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:57.250647    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7095-82ff58" podUID="ed345c07-8bf9-4c64-b755-5b37e6198aba" containerName="test-curl-7095-82ff58" containerID="cri-o://a25d35b26c6553d8d30042dba04568f3de77c74c8ecc0682993e0f3f689b39a5" gracePeriod=2
Nov 03 19:48:57.258751 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:48:57.257162    5237 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7095-82ff58\" is forbidden: User \"system:node:ip-10-0-133-29.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-29.us-east-2.compute.internal' and this object" podUID="ed345c07-8bf9-4c64-b755-5b37e6198aba" pod="opendatahub/test-curl-7095-82ff58"
Nov 03 19:48:57.811536 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:57.811499    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:58.833814 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:58.833769    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:59.524266 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:59.524196    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-82ff58"
Nov 03 19:48:59.547874 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:59.547826    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed345c07-8bf9-4c64-b755-5b37e6198aba-kube-api-access-7gl9r\" (UniqueName: \"kubernetes.io/projected/ed345c07-8bf9-4c64-b755-5b37e6198aba-kube-api-access-7gl9r\") pod \"ed345c07-8bf9-4c64-b755-5b37e6198aba\" (UID: \"ed345c07-8bf9-4c64-b755-5b37e6198aba\") "
Nov 03 19:48:59.550156 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:59.550127    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed345c07-8bf9-4c64-b755-5b37e6198aba-kube-api-access-7gl9r" pod "ed345c07-8bf9-4c64-b755-5b37e6198aba" (UID: "ed345c07-8bf9-4c64-b755-5b37e6198aba"). InnerVolumeSpecName "kube-api-access-7gl9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:59.650519 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:59.650478    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gl9r\" (UniqueName: \"kubernetes.io/projected/ed345c07-8bf9-4c64-b755-5b37e6198aba-kube-api-access-7gl9r\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:59.708040 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:59.707816    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:59.876091 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:48:59.876053    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed345c07-8bf9-4c64-b755-5b37e6198aba" path="/var/lib/kubelet/pods/ed345c07-8bf9-4c64-b755-5b37e6198aba/volumes"
Nov 03 19:49:00.348534 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:00.348316    5237 generic.go:356] "Generic (PLEG): container finished" podID="ed345c07-8bf9-4c64-b755-5b37e6198aba" containerID="a25d35b26c6553d8d30042dba04568f3de77c74c8ecc0682993e0f3f689b39a5" exitCode=137
Nov 03 19:49:00.348534 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:00.348372    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-82ff58"
Nov 03 19:49:00.348534 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:00.348390    5237 scope.go:122] "RemoveContainer" containerID="a25d35b26c6553d8d30042dba04568f3de77c74c8ecc0682993e0f3f689b39a5"
Nov 03 19:49:00.359389 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:00.359367    5237 scope.go:122] "RemoveContainer" containerID="a25d35b26c6553d8d30042dba04568f3de77c74c8ecc0682993e0f3f689b39a5"
Nov 03 19:49:00.359725 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:49:00.359695    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25d35b26c6553d8d30042dba04568f3de77c74c8ecc0682993e0f3f689b39a5\": container with ID starting with a25d35b26c6553d8d30042dba04568f3de77c74c8ecc0682993e0f3f689b39a5 not found: ID does not exist" containerID="a25d35b26c6553d8d30042dba04568f3de77c74c8ecc0682993e0f3f689b39a5"
Nov 03 19:49:00.359877 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:00.359732    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25d35b26c6553d8d30042dba04568f3de77c74c8ecc0682993e0f3f689b39a5"} err="failed to get container status \"a25d35b26c6553d8d30042dba04568f3de77c74c8ecc0682993e0f3f689b39a5\": rpc error: code = NotFound desc = could not find container \"a25d35b26c6553d8d30042dba04568f3de77c74c8ecc0682993e0f3f689b39a5\": container with ID starting with a25d35b26c6553d8d30042dba04568f3de77c74c8ecc0682993e0f3f689b39a5 not found: ID does not exist"
Nov 03 19:49:00.443764 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:00.443728    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:00.652825 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:00.652675    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:00.788808 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:00.788767    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:01.353527 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:01.352790    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:01.876815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:01.876776    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:01.907552 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:01.907515    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"]
Nov 03 19:49:01.908347 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:01.908327    5237 cpu_manager.go:414] "containerMap: removing container" podUID="ed345c07-8bf9-4c64-b755-5b37e6198aba" containerName="test-curl-7095-82ff58"
Nov 03 19:49:01.908347 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:01.908347    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed345c07-8bf9-4c64-b755-5b37e6198aba" containerName="test-curl-7095-82ff58"
Nov 03 19:49:01.908492 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:01.908481    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed345c07-8bf9-4c64-b755-5b37e6198aba" containerName="test-curl-7095-82ff58"
Nov 03 19:49:01.914444 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:01.914420    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"
Nov 03 19:49:01.921543 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:01.921481    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-10e551\""
Nov 03 19:49:01.925930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:01.925902    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"]
Nov 03 19:49:01.926140 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:01.926125    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-10e551-dockercfg-7gndr\""
Nov 03 19:49:02.019666 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:02.019623    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:02.094864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:02.065375    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf3379f5-4f44-43e1-beb2-9ea9b3511be1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr\" (UID: \"cf3379f5-4f44-43e1-beb2-9ea9b3511be1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"
Nov 03 19:49:02.095637 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:02.095595    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd9xd\" (UniqueName: \"kubernetes.io/projected/cf3379f5-4f44-43e1-beb2-9ea9b3511be1-kube-api-access-dd9xd\") pod \"payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr\" (UID: \"cf3379f5-4f44-43e1-beb2-9ea9b3511be1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"
Nov 03 19:49:02.204361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:02.204261    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd9xd\" (UniqueName: \"kubernetes.io/projected/cf3379f5-4f44-43e1-beb2-9ea9b3511be1-kube-api-access-dd9xd\") pod \"payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr\" (UID: \"cf3379f5-4f44-43e1-beb2-9ea9b3511be1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"
Nov 03 19:49:02.205813 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:02.205783    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf3379f5-4f44-43e1-beb2-9ea9b3511be1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr\" (UID: \"cf3379f5-4f44-43e1-beb2-9ea9b3511be1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"
Nov 03 19:49:02.239853 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:02.229615    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf3379f5-4f44-43e1-beb2-9ea9b3511be1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr\" (UID: \"cf3379f5-4f44-43e1-beb2-9ea9b3511be1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"
Nov 03 19:49:02.421464 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:02.421371    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd9xd\" (UniqueName: \"kubernetes.io/projected/cf3379f5-4f44-43e1-beb2-9ea9b3511be1-kube-api-access-dd9xd\") pod \"payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr\" (UID: \"cf3379f5-4f44-43e1-beb2-9ea9b3511be1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"
Nov 03 19:49:02.536651 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:02.536626    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"
Nov 03 19:49:02.625304 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:02.625199    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:02.891773 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:02.891742    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"]
Nov 03 19:49:02.892116 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:49:02.892087    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3379f5_4f44_43e1_beb2_9ea9b3511be1.slice/crio-657986b624ea9c406a91cabfea19f9927c9c696a7cd87869a464a28f9b11c5dc WatchSource:0}: Error finding container 657986b624ea9c406a91cabfea19f9927c9c696a7cd87869a464a28f9b11c5dc: Status 404 returned error can't find the container with id 657986b624ea9c406a91cabfea19f9927c9c696a7cd87869a464a28f9b11c5dc
Nov 03 19:49:03.199066 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:03.198979    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:03.414003 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:03.413965    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr" event={"ID":"cf3379f5-4f44-43e1-beb2-9ea9b3511be1","Type":"ContainerStarted","Data":"d6c58eed3fa08f80bb1f5ee4905b2211736461d33815b3ed231da688b58149ce"}
Nov 03 19:49:03.414286 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:03.414029    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr" event={"ID":"cf3379f5-4f44-43e1-beb2-9ea9b3511be1","Type":"ContainerStarted","Data":"657986b624ea9c406a91cabfea19f9927c9c696a7cd87869a464a28f9b11c5dc"}
Nov 03 19:49:03.414286 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:03.414198    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"
Nov 03 19:49:03.438109 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:03.438040    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr" podStartSLOduration=2.4380205 podStartE2EDuration="2.4380205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:03.433085951 +0000 UTC m=+1299.687393184" watchObservedRunningTime="2026-11-03 19:49:03.4380205 +0000 UTC m=+1299.692327730"
Nov 03 19:49:03.918601 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:03.918553    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:04.643097 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:04.643052    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:05.328638 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:05.328604    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:05.954198 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:05.953827    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:06.571870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:06.565929    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:06.732602 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:06.723185    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:07.438614 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:07.438578    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:08.139049 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:08.139009    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:08.997027 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:08.996975    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:09.817641 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:09.817611    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:10.567871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:10.567531    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:10.762570 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:10.762531    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"]
Nov 03 19:49:10.762846 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:10.762801    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" podUID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" containerName="istio-proxy" containerID="cri-o://49be4943cff4db862d760349b0104e187f0fae677588243f65ca07484e37f037" gracePeriod=30
Nov 03 19:49:11.526894 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:11.526858    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:12.107928 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:12.107787    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:12.297676 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:12.297643    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:12.431457 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:12.431356    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:13.166858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:13.166391    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:13.530909 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:49:13.530642    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:49:14.112216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:14.112177    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:14.427415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:14.427338    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"
Nov 03 19:49:14.902314 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:14.902277    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:15.541124 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:15.541084    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:15.722158 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:15.722122    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:15.912959 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:15.912925    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:16.128190 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.128105    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:16.310330 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.310295    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:16.582859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.578044    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:16.611474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.611437    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:49:16.684573 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.684533    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-credential-socket\") pod \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") "
Nov 03 19:49:16.685251 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.685215    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-token\" (UniqueName: \"kubernetes.io/projected/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-token\") pod \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") "
Nov 03 19:49:16.686098 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.686063    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-podinfo\") pod \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") "
Nov 03 19:49:16.686326 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.686297    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-envoy\") pod \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") "
Nov 03 19:49:16.686537 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.686517    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-workload-socket\") pod \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") "
Nov 03 19:49:16.686771 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.686755    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-workload-certs\") pod \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") "
Nov 03 19:49:16.687063 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.687042    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-kube-api-access-rjpfs\" (UniqueName: \"kubernetes.io/projected/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-kube-api-access-rjpfs\") pod \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") "
Nov 03 19:49:16.687279 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.687262    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-data\") pod \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") "
Nov 03 19:49:16.687467 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.687440    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istiod-ca-cert\") pod \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\" (UID: \"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7\") "
Nov 03 19:49:16.690717 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.685110    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-credential-socket" pod "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" (UID: "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:49:16.690717 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.690319    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-workload-socket" pod "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" (UID: "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:49:16.691215 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.691051    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-data" pod "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" (UID: "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:49:16.691406 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.691372    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-workload-certs" pod "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" (UID: "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:49:16.710032 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.701144    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istiod-ca-cert" pod "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" (UID: "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:49:16.742416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.731645    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-envoy" pod "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" (UID: "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:49:16.744248 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.744202    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:16.754495 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.748806    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-podinfo" pod "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" (UID: "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:49:16.754495 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.754033    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-token" pod "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" (UID: "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:49:16.795868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.792098    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-kube-api-access-rjpfs" pod "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" (UID: "33872d2d-bbf6-4242-a6c3-a0f3fffb54f7"). InnerVolumeSpecName "kube-api-access-rjpfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:49:16.795868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.793596    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjpfs\" (UniqueName: \"kubernetes.io/projected/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-kube-api-access-rjpfs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:16.795868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.793628    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:16.795868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.793642    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:16.795868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.793656    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:16.795868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.793670    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:16.795868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.793683    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:16.795868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.793696    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:16.795868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.793709    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:16.795868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.793722    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:16.861599 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.861512    5237 generic.go:356] "Generic (PLEG): container finished" podID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" containerID="49be4943cff4db862d760349b0104e187f0fae677588243f65ca07484e37f037" exitCode=0
Nov 03 19:49:16.863057 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.861867    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"
Nov 03 19:49:16.867668 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.861894    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" event={"ID":"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7","Type":"ContainerDied","Data":"49be4943cff4db862d760349b0104e187f0fae677588243f65ca07484e37f037"}
Nov 03 19:49:16.867928 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.867905    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl" event={"ID":"33872d2d-bbf6-4242-a6c3-a0f3fffb54f7","Type":"ContainerDied","Data":"f63840653d8a05432cd381f43d8479847e76dd3d24443206b7f974f3a932069f"}
Nov 03 19:49:16.868113 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.868070    5237 scope.go:122] "RemoveContainer" containerID="49be4943cff4db862d760349b0104e187f0fae677588243f65ca07484e37f037"
Nov 03 19:49:16.884829 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.884803    5237 scope.go:122] "RemoveContainer" containerID="49be4943cff4db862d760349b0104e187f0fae677588243f65ca07484e37f037"
Nov 03 19:49:16.885239 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:49:16.885212    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49be4943cff4db862d760349b0104e187f0fae677588243f65ca07484e37f037\": container with ID starting with 49be4943cff4db862d760349b0104e187f0fae677588243f65ca07484e37f037 not found: ID does not exist" containerID="49be4943cff4db862d760349b0104e187f0fae677588243f65ca07484e37f037"
Nov 03 19:49:16.885308 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.885253    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49be4943cff4db862d760349b0104e187f0fae677588243f65ca07484e37f037"} err="failed to get container status \"49be4943cff4db862d760349b0104e187f0fae677588243f65ca07484e37f037\": rpc error: code = NotFound desc = could not find container \"49be4943cff4db862d760349b0104e187f0fae677588243f65ca07484e37f037\": container with ID starting with 49be4943cff4db862d760349b0104e187f0fae677588243f65ca07484e37f037 not found: ID does not exist"
Nov 03 19:49:16.925214 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:16.925180    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"]
Nov 03 19:49:17.019861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:17.015088    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:17.036072 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:17.033430    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-gcxcl"]
Nov 03 19:49:17.721041 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:17.717120    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:17.864691 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:17.864655    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" path="/var/lib/kubelet/pods/33872d2d-bbf6-4242-a6c3-a0f3fffb54f7/volumes"
Nov 03 19:49:18.460615 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:18.460572    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:18.869933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:18.869896    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"]
Nov 03 19:49:18.962194 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:18.944154    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"]
Nov 03 19:49:18.970647 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:18.970552    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:19.122957 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:19.122863    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:19.677037 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:19.676986    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:20.272555 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:20.272519    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:20.440389 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:20.440349    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:20.882328 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:20.882260    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:21.079943 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:21.079911    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:21.349089 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:21.349046    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:21.520859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:21.515971    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:21.733130 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:21.733046    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"]
Nov 03 19:49:21.879733 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:21.879688    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:22.247980 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:22.247929    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:22.442552 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:22.442500    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:22.638220 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:22.638141    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:23.362013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:23.361821    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:23.933798 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:23.933761    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:23.958976 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:23.939774    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp" podUID="e80c1008-2bd2-4a10-bacf-ec1d7148ebf8" containerName="payload-processing" containerID="cri-o://2e406443618c8fb1896643c1b0aab402a0a83c70ea059d128c8734d02fbaab42" gracePeriod=25
Nov 03 19:49:23.978826 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:23.978788    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk" podUID="7f80a141-ae01-4ca9-911f-f2fe09d5492b" containerName="payload-pre-processing" containerID="cri-o://40e9571065bb0d9e5f721144659703671bb70f494e2172efdf9eaa04e75c66cb" gracePeriod=25
Nov 03 19:49:24.313769 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:24.313740    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"
Nov 03 19:49:24.366369 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:24.366335    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"
Nov 03 19:49:24.484377 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:24.484278    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e80c1008-2bd2-4a10-bacf-ec1d7148ebf8-kube-api-access-jjnrn\" (UniqueName: \"kubernetes.io/projected/e80c1008-2bd2-4a10-bacf-ec1d7148ebf8-kube-api-access-jjnrn\") pod \"e80c1008-2bd2-4a10-bacf-ec1d7148ebf8\" (UID: \"e80c1008-2bd2-4a10-bacf-ec1d7148ebf8\") "
Nov 03 19:49:24.484563 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:24.484443    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e80c1008-2bd2-4a10-bacf-ec1d7148ebf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80c1008-2bd2-4a10-bacf-ec1d7148ebf8-plugins-config-volume\") pod \"e80c1008-2bd2-4a10-bacf-ec1d7148ebf8\" (UID: \"e80c1008-2bd2-4a10-bacf-ec1d7148ebf8\") "
Nov 03 19:49:24.484659 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:24.484642    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f80a141-ae01-4ca9-911f-f2fe09d5492b-kube-api-access-xhzc6\" (UniqueName: \"kubernetes.io/projected/7f80a141-ae01-4ca9-911f-f2fe09d5492b-kube-api-access-xhzc6\") pod \"7f80a141-ae01-4ca9-911f-f2fe09d5492b\" (UID: \"7f80a141-ae01-4ca9-911f-f2fe09d5492b\") "
Nov 03 19:49:24.484865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:24.484807    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80c1008-2bd2-4a10-bacf-ec1d7148ebf8-plugins-config-volume" pod "e80c1008-2bd2-4a10-bacf-ec1d7148ebf8" (UID: "e80c1008-2bd2-4a10-bacf-ec1d7148ebf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:49:24.484865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:24.484850    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f80a141-ae01-4ca9-911f-f2fe09d5492b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f80a141-ae01-4ca9-911f-f2fe09d5492b-plugins-config-volume\") pod \"7f80a141-ae01-4ca9-911f-f2fe09d5492b\" (UID: \"7f80a141-ae01-4ca9-911f-f2fe09d5492b\") "
Nov 03 19:49:24.485569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:24.485285    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f80a141-ae01-4ca9-911f-f2fe09d5492b-plugins-config-volume" pod "7f80a141-ae01-4ca9-911f-f2fe09d5492b" (UID: "7f80a141-ae01-4ca9-911f-f2fe09d5492b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:49:24.485569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:24.485533    5237 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80c1008-2bd2-4a10-bacf-ec1d7148ebf8-plugins-config-volume\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:24.485569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:24.485550    5237 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f80a141-ae01-4ca9-911f-f2fe09d5492b-plugins-config-volume\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:24.486771 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:24.486747    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80c1008-2bd2-4a10-bacf-ec1d7148ebf8-kube-api-access-jjnrn" pod "e80c1008-2bd2-4a10-bacf-ec1d7148ebf8" (UID: "e80c1008-2bd2-4a10-bacf-ec1d7148ebf8"). InnerVolumeSpecName "kube-api-access-jjnrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:49:24.486981 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:24.486963    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f80a141-ae01-4ca9-911f-f2fe09d5492b-kube-api-access-xhzc6" pod "7f80a141-ae01-4ca9-911f-f2fe09d5492b" (UID: "7f80a141-ae01-4ca9-911f-f2fe09d5492b"). InnerVolumeSpecName "kube-api-access-xhzc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:49:24.558727 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:24.558667    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:24.683907 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:24.665541    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhzc6\" (UniqueName: \"kubernetes.io/projected/7f80a141-ae01-4ca9-911f-f2fe09d5492b-kube-api-access-xhzc6\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:24.683907 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:24.671144    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjnrn\" (UniqueName: \"kubernetes.io/projected/e80c1008-2bd2-4a10-bacf-ec1d7148ebf8-kube-api-access-jjnrn\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:25.041661 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.041629    5237 generic.go:356] "Generic (PLEG): container finished" podID="e80c1008-2bd2-4a10-bacf-ec1d7148ebf8" containerID="2e406443618c8fb1896643c1b0aab402a0a83c70ea059d128c8734d02fbaab42" exitCode=0
Nov 03 19:49:25.042136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.041685    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"
Nov 03 19:49:25.042136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.041713    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp" event={"ID":"e80c1008-2bd2-4a10-bacf-ec1d7148ebf8","Type":"ContainerDied","Data":"2e406443618c8fb1896643c1b0aab402a0a83c70ea059d128c8734d02fbaab42"}
Nov 03 19:49:25.042136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.041774    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp" event={"ID":"e80c1008-2bd2-4a10-bacf-ec1d7148ebf8","Type":"ContainerDied","Data":"0e4a6cc4452195b020e5875ec70bef5cf69bec838c459887f8d4dfa1fd38c610"}
Nov 03 19:49:25.042136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.041803    5237 scope.go:122] "RemoveContainer" containerID="2e406443618c8fb1896643c1b0aab402a0a83c70ea059d128c8734d02fbaab42"
Nov 03 19:49:25.043090 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.043069    5237 generic.go:356] "Generic (PLEG): container finished" podID="7f80a141-ae01-4ca9-911f-f2fe09d5492b" containerID="40e9571065bb0d9e5f721144659703671bb70f494e2172efdf9eaa04e75c66cb" exitCode=0
Nov 03 19:49:25.043164 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.043133    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"
Nov 03 19:49:25.043208 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.043164    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk" event={"ID":"7f80a141-ae01-4ca9-911f-f2fe09d5492b","Type":"ContainerDied","Data":"40e9571065bb0d9e5f721144659703671bb70f494e2172efdf9eaa04e75c66cb"}
Nov 03 19:49:25.043254 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.043210    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk" event={"ID":"7f80a141-ae01-4ca9-911f-f2fe09d5492b","Type":"ContainerDied","Data":"8045e27005b6e00d936e646c0ca7647dfa6cd08358ac48c86c634ddcbb25e91f"}
Nov 03 19:49:25.054508 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.054487    5237 scope.go:122] "RemoveContainer" containerID="2e406443618c8fb1896643c1b0aab402a0a83c70ea059d128c8734d02fbaab42"
Nov 03 19:49:25.054767 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:49:25.054748    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e406443618c8fb1896643c1b0aab402a0a83c70ea059d128c8734d02fbaab42\": container with ID starting with 2e406443618c8fb1896643c1b0aab402a0a83c70ea059d128c8734d02fbaab42 not found: ID does not exist" containerID="2e406443618c8fb1896643c1b0aab402a0a83c70ea059d128c8734d02fbaab42"
Nov 03 19:49:25.054870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.054778    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e406443618c8fb1896643c1b0aab402a0a83c70ea059d128c8734d02fbaab42"} err="failed to get container status \"2e406443618c8fb1896643c1b0aab402a0a83c70ea059d128c8734d02fbaab42\": rpc error: code = NotFound desc = could not find container \"2e406443618c8fb1896643c1b0aab402a0a83c70ea059d128c8734d02fbaab42\": container with ID starting with 2e406443618c8fb1896643c1b0aab402a0a83c70ea059d128c8734d02fbaab42 not found: ID does not exist"
Nov 03 19:49:25.054870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.054797    5237 scope.go:122] "RemoveContainer" containerID="40e9571065bb0d9e5f721144659703671bb70f494e2172efdf9eaa04e75c66cb"
Nov 03 19:49:25.064311 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.064289    5237 scope.go:122] "RemoveContainer" containerID="40e9571065bb0d9e5f721144659703671bb70f494e2172efdf9eaa04e75c66cb"
Nov 03 19:49:25.064636 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:49:25.064567    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e9571065bb0d9e5f721144659703671bb70f494e2172efdf9eaa04e75c66cb\": container with ID starting with 40e9571065bb0d9e5f721144659703671bb70f494e2172efdf9eaa04e75c66cb not found: ID does not exist" containerID="40e9571065bb0d9e5f721144659703671bb70f494e2172efdf9eaa04e75c66cb"
Nov 03 19:49:25.064636 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.064602    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e9571065bb0d9e5f721144659703671bb70f494e2172efdf9eaa04e75c66cb"} err="failed to get container status \"40e9571065bb0d9e5f721144659703671bb70f494e2172efdf9eaa04e75c66cb\": rpc error: code = NotFound desc = could not find container \"40e9571065bb0d9e5f721144659703671bb70f494e2172efdf9eaa04e75c66cb\": container with ID starting with 40e9571065bb0d9e5f721144659703671bb70f494e2172efdf9eaa04e75c66cb not found: ID does not exist"
Nov 03 19:49:25.068189 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.068168    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"]
Nov 03 19:49:25.073515 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.073494    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f292e03c-7b49c666cf-cbdbp"]
Nov 03 19:49:25.085595 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.085552    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"]
Nov 03 19:49:25.089617 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.089591    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f292e03c-6dbbff8f95-s4hlk"]
Nov 03 19:49:25.263605 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.263571    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:25.862937 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.862901    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f80a141-ae01-4ca9-911f-f2fe09d5492b" path="/var/lib/kubelet/pods/7f80a141-ae01-4ca9-911f-f2fe09d5492b/volumes"
Nov 03 19:49:25.863422 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.863404    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e80c1008-2bd2-4a10-bacf-ec1d7148ebf8" path="/var/lib/kubelet/pods/e80c1008-2bd2-4a10-bacf-ec1d7148ebf8/volumes"
Nov 03 19:49:25.938773 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:25.938722    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:26.572073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:26.570938    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:27.295427 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:27.280868    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:27.517160 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:27.517122    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:27.768203 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:27.768101    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:28.508812 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:28.508772    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:29.014454 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:29.014416    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:29.150306 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:29.150271    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:29.369188 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:29.369152    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:29.523728 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:29.523691    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:30.112608 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:30.112562    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:30.247227 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:30.247194    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:30.493745 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:30.493636    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:30.765636 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:30.765113    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:30.921156 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:30.921121    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:31.129253 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:31.126138    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:31.271858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:31.270965    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:31.826202 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:31.826163    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:32.141529 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:32.141448    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:32.503584 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:32.503492    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:32.692082 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:32.692034    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:33.264879 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:33.264789    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:34.357503 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:34.357428    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:34.495433 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:34.495397    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:34.630591 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:34.630497    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:34.773975 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:34.773936    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:34.911792 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:34.911697    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:35.749319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:35.749279    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:36.157111 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:36.157076    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:36.291688 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:36.291614    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:36.681898 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:36.680713    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:36.755402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:36.755359    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6" podUID="568c2367-6f5c-423f-9c5b-5cc760e4cb7b" containerName="main" containerID="cri-o://9b637d2afd0d34cd5e58421a8d27b7cbb008af86fcc9cc7fdce82f17afa3eb93" gracePeriod=45
Nov 03 19:49:36.954708 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:36.954649    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:37.043031 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.043003    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:49:37.109521 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.109478    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-tls-certs\" (UniqueName: \"kubernetes.io/secret/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-tls-certs\") pod \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") "
Nov 03 19:49:37.109717 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.109572    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:37.109717 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.109615    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-model-cache\") pod \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") "
Nov 03 19:49:37.109717 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.109671    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-home\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-home\") pod \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") "
Nov 03 19:49:37.109949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.109779    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-dshm\") pod \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") "
Nov 03 19:49:37.109949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.109925    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-tmp-dir\") pod \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") "
Nov 03 19:49:37.110059 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.109965    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-kube-api-access-ghxt6\" (UniqueName: \"kubernetes.io/projected/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-kube-api-access-ghxt6\") pod \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\" (UID: \"568c2367-6f5c-423f-9c5b-5cc760e4cb7b\") "
Nov 03 19:49:37.110512 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.110134    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-model-cache" pod "568c2367-6f5c-423f-9c5b-5cc760e4cb7b" (UID: "568c2367-6f5c-423f-9c5b-5cc760e4cb7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:49:37.110702 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.110684    5237 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-model-cache\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:37.111659 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.111616    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-home" pod "568c2367-6f5c-423f-9c5b-5cc760e4cb7b" (UID: "568c2367-6f5c-423f-9c5b-5cc760e4cb7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:49:37.111854 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.111819    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-tmp-dir" pod "568c2367-6f5c-423f-9c5b-5cc760e4cb7b" (UID: "568c2367-6f5c-423f-9c5b-5cc760e4cb7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:49:37.112947 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.112905    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-tls-certs" pod "568c2367-6f5c-423f-9c5b-5cc760e4cb7b" (UID: "568c2367-6f5c-423f-9c5b-5cc760e4cb7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:49:37.114771 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.114733    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-dshm" pod "568c2367-6f5c-423f-9c5b-5cc760e4cb7b" (UID: "568c2367-6f5c-423f-9c5b-5cc760e4cb7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:49:37.119691 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.119664    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-kube-api-access-ghxt6" pod "568c2367-6f5c-423f-9c5b-5cc760e4cb7b" (UID: "568c2367-6f5c-423f-9c5b-5cc760e4cb7b"). InnerVolumeSpecName "kube-api-access-ghxt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:49:37.212667 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.212578    5237 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-tls-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:37.212667 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.212618    5237 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-home\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:37.212667 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.212631    5237 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-dshm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:37.212667 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.212643    5237 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-tmp-dir\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:37.212667 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.212658    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghxt6\" (UniqueName: \"kubernetes.io/projected/568c2367-6f5c-423f-9c5b-5cc760e4cb7b-kube-api-access-ghxt6\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:37.382724 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.380776    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:37.708764 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.708727    5237 generic.go:356] "Generic (PLEG): container finished" podID="568c2367-6f5c-423f-9c5b-5cc760e4cb7b" containerID="9b637d2afd0d34cd5e58421a8d27b7cbb008af86fcc9cc7fdce82f17afa3eb93" exitCode=0
Nov 03 19:49:37.709000 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.708773    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"
Nov 03 19:49:37.709000 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.708821    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6" event={"ID":"568c2367-6f5c-423f-9c5b-5cc760e4cb7b","Type":"ContainerDied","Data":"9b637d2afd0d34cd5e58421a8d27b7cbb008af86fcc9cc7fdce82f17afa3eb93"}
Nov 03 19:49:37.709000 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.708881    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6" event={"ID":"568c2367-6f5c-423f-9c5b-5cc760e4cb7b","Type":"ContainerDied","Data":"f9603ac8f443daf7e61f8b6088988afe84e968a8a2e07d49d324b6a78543bd17"}
Nov 03 19:49:37.709000 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.708899    5237 scope.go:122] "RemoveContainer" containerID="9b637d2afd0d34cd5e58421a8d27b7cbb008af86fcc9cc7fdce82f17afa3eb93"
Nov 03 19:49:37.720454 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.720429    5237 scope.go:122] "RemoveContainer" containerID="9b637d2afd0d34cd5e58421a8d27b7cbb008af86fcc9cc7fdce82f17afa3eb93"
Nov 03 19:49:37.720746 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:49:37.720726    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b637d2afd0d34cd5e58421a8d27b7cbb008af86fcc9cc7fdce82f17afa3eb93\": container with ID starting with 9b637d2afd0d34cd5e58421a8d27b7cbb008af86fcc9cc7fdce82f17afa3eb93 not found: ID does not exist" containerID="9b637d2afd0d34cd5e58421a8d27b7cbb008af86fcc9cc7fdce82f17afa3eb93"
Nov 03 19:49:37.720830 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.720753    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b637d2afd0d34cd5e58421a8d27b7cbb008af86fcc9cc7fdce82f17afa3eb93"} err="failed to get container status \"9b637d2afd0d34cd5e58421a8d27b7cbb008af86fcc9cc7fdce82f17afa3eb93\": rpc error: code = NotFound desc = could not find container \"9b637d2afd0d34cd5e58421a8d27b7cbb008af86fcc9cc7fdce82f17afa3eb93\": container with ID starting with 9b637d2afd0d34cd5e58421a8d27b7cbb008af86fcc9cc7fdce82f17afa3eb93 not found: ID does not exist"
Nov 03 19:49:37.734444 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.734406    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"]
Nov 03 19:49:37.737902 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.737870    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-eae28ff0/e2e-lifecycle-model-eae28ff0-kserve-57f6b4b9bf-mdtg6"]
Nov 03 19:49:37.861522 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.861485    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="568c2367-6f5c-423f-9c5b-5cc760e4cb7b" path="/var/lib/kubelet/pods/568c2367-6f5c-423f-9c5b-5cc760e4cb7b/volumes"
Nov 03 19:49:37.993501 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:37.993406    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:38.138787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:38.138748    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:38.314941 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:38.307386    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:38.603458 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:38.603309    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:39.071547 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:39.071507    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:39.835451 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:39.835417    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:40.483037 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:40.483000    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:40.808106 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:40.808074    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"]
Nov 03 19:49:40.808402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:40.808378    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk" podUID="1c377e40-757e-47a6-b015-bf914f369081" containerName="istio-proxy" containerID="cri-o://56934bdc39df6cff78c53eba2f07cb66fe21aeebde40e4bc4a4ed245a5c607e7" gracePeriod=30
Nov 03 19:49:41.271956 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:41.260389    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:42.188128 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:42.188077    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:43.035200 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:43.035156    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:43.282897 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:49:43.282822    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:49:43.730099 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:43.730066    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:44.398391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:44.397109    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:45.037593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:45.037562    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:45.639946 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:45.639901    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:45.771345 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:45.771303    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:46.092641 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.092615    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:49:46.224541 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.224460    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-istio-envoy\") pod \"1c377e40-757e-47a6-b015-bf914f369081\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") "
Nov 03 19:49:46.224701 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.224615    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c377e40-757e-47a6-b015-bf914f369081-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c377e40-757e-47a6-b015-bf914f369081-istiod-ca-cert\") pod \"1c377e40-757e-47a6-b015-bf914f369081\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") "
Nov 03 19:49:46.224701 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.224641    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c377e40-757e-47a6-b015-bf914f369081-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c377e40-757e-47a6-b015-bf914f369081-istio-podinfo\") pod \"1c377e40-757e-47a6-b015-bf914f369081\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") "
Nov 03 19:49:46.224701 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.224679    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-workload-certs\") pod \"1c377e40-757e-47a6-b015-bf914f369081\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") "
Nov 03 19:49:46.224926 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.224721    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c377e40-757e-47a6-b015-bf914f369081-istio-token\" (UniqueName: \"kubernetes.io/projected/1c377e40-757e-47a6-b015-bf914f369081-istio-token\") pod \"1c377e40-757e-47a6-b015-bf914f369081\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") "
Nov 03 19:49:46.224926 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.224781    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-credential-socket\") pod \"1c377e40-757e-47a6-b015-bf914f369081\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") "
Nov 03 19:49:46.225054 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.225024    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-workload-certs" pod "1c377e40-757e-47a6-b015-bf914f369081" (UID: "1c377e40-757e-47a6-b015-bf914f369081"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:49:46.225135 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.225113    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c377e40-757e-47a6-b015-bf914f369081-istiod-ca-cert" pod "1c377e40-757e-47a6-b015-bf914f369081" (UID: "1c377e40-757e-47a6-b015-bf914f369081"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:49:46.225184 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.225115    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-credential-socket" pod "1c377e40-757e-47a6-b015-bf914f369081" (UID: "1c377e40-757e-47a6-b015-bf914f369081"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:49:46.225236 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.225189    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-workload-socket\") pod \"1c377e40-757e-47a6-b015-bf914f369081\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") "
Nov 03 19:49:46.225284 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.225248    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-istio-data\") pod \"1c377e40-757e-47a6-b015-bf914f369081\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") "
Nov 03 19:49:46.225435 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.225341    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c377e40-757e-47a6-b015-bf914f369081-kube-api-access-x4vcz\" (UniqueName: \"kubernetes.io/projected/1c377e40-757e-47a6-b015-bf914f369081-kube-api-access-x4vcz\") pod \"1c377e40-757e-47a6-b015-bf914f369081\" (UID: \"1c377e40-757e-47a6-b015-bf914f369081\") "
Nov 03 19:49:46.225867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.225536    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-workload-socket" pod "1c377e40-757e-47a6-b015-bf914f369081" (UID: "1c377e40-757e-47a6-b015-bf914f369081"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:49:46.225867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.225641    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-istio-data" pod "1c377e40-757e-47a6-b015-bf914f369081" (UID: "1c377e40-757e-47a6-b015-bf914f369081"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:49:46.239807 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.238421    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c377e40-757e-47a6-b015-bf914f369081-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:46.239807 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.238457    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:46.239807 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.238475    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:46.239807 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.239730    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:46.239807 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.239751    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:46.242747 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.242501    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-istio-envoy" pod "1c377e40-757e-47a6-b015-bf914f369081" (UID: "1c377e40-757e-47a6-b015-bf914f369081"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:49:46.242747 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.242540    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c377e40-757e-47a6-b015-bf914f369081-istio-token" pod "1c377e40-757e-47a6-b015-bf914f369081" (UID: "1c377e40-757e-47a6-b015-bf914f369081"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:49:46.242747 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.242650    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c377e40-757e-47a6-b015-bf914f369081-kube-api-access-x4vcz" pod "1c377e40-757e-47a6-b015-bf914f369081" (UID: "1c377e40-757e-47a6-b015-bf914f369081"). InnerVolumeSpecName "kube-api-access-x4vcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:49:46.246390 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.246296    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c377e40-757e-47a6-b015-bf914f369081-istio-podinfo" pod "1c377e40-757e-47a6-b015-bf914f369081" (UID: "1c377e40-757e-47a6-b015-bf914f369081"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:49:46.326718 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.326683    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:46.341161 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.341131    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c377e40-757e-47a6-b015-bf914f369081-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:46.341161 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.341159    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c377e40-757e-47a6-b015-bf914f369081-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:46.341368 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.341168    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4vcz\" (UniqueName: \"kubernetes.io/projected/1c377e40-757e-47a6-b015-bf914f369081-kube-api-access-x4vcz\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:46.341368 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.341179    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c377e40-757e-47a6-b015-bf914f369081-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:46.835029 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.834996    5237 generic.go:356] "Generic (PLEG): container finished" podID="1c377e40-757e-47a6-b015-bf914f369081" containerID="56934bdc39df6cff78c53eba2f07cb66fe21aeebde40e4bc4a4ed245a5c607e7" exitCode=0
Nov 03 19:49:46.836052 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.835044    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"
Nov 03 19:49:46.836052 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.835084    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk" event={"ID":"1c377e40-757e-47a6-b015-bf914f369081","Type":"ContainerDied","Data":"56934bdc39df6cff78c53eba2f07cb66fe21aeebde40e4bc4a4ed245a5c607e7"}
Nov 03 19:49:46.836052 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.835143    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk" event={"ID":"1c377e40-757e-47a6-b015-bf914f369081","Type":"ContainerDied","Data":"5d9e0c734b7f6d3da0e1b6f4ad871a424bfb765bdac36a04b938a1288a246b26"}
Nov 03 19:49:46.836052 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.835174    5237 scope.go:122] "RemoveContainer" containerID="56934bdc39df6cff78c53eba2f07cb66fe21aeebde40e4bc4a4ed245a5c607e7"
Nov 03 19:49:46.847007 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.846987    5237 scope.go:122] "RemoveContainer" containerID="56934bdc39df6cff78c53eba2f07cb66fe21aeebde40e4bc4a4ed245a5c607e7"
Nov 03 19:49:46.847335 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:49:46.847313    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56934bdc39df6cff78c53eba2f07cb66fe21aeebde40e4bc4a4ed245a5c607e7\": container with ID starting with 56934bdc39df6cff78c53eba2f07cb66fe21aeebde40e4bc4a4ed245a5c607e7 not found: ID does not exist" containerID="56934bdc39df6cff78c53eba2f07cb66fe21aeebde40e4bc4a4ed245a5c607e7"
Nov 03 19:49:46.847384 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.847346    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56934bdc39df6cff78c53eba2f07cb66fe21aeebde40e4bc4a4ed245a5c607e7"} err="failed to get container status \"56934bdc39df6cff78c53eba2f07cb66fe21aeebde40e4bc4a4ed245a5c607e7\": rpc error: code = NotFound desc = could not find container \"56934bdc39df6cff78c53eba2f07cb66fe21aeebde40e4bc4a4ed245a5c607e7\": container with ID starting with 56934bdc39df6cff78c53eba2f07cb66fe21aeebde40e4bc4a4ed245a5c607e7 not found: ID does not exist"
Nov 03 19:49:46.862732 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.862704    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"]
Nov 03 19:49:46.867966 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.867942    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6rqhk"]
Nov 03 19:49:46.944065 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:46.944029    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:47.685096 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:47.685057    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:47.864355 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:47.864321    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c377e40-757e-47a6-b015-bf914f369081" path="/var/lib/kubelet/pods/1c377e40-757e-47a6-b015-bf914f369081/volumes"
Nov 03 19:49:47.871945 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:47.871917    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:48.006129 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:48.006039    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:48.633165 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:48.633128    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:49.623260 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:49.623161    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:49.776058 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:49.776019    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:50.862858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:50.855863    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:51.627747 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:51.627703    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:51.827264 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:51.827225    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:52.110722 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:52.110689    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:52.421154 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:52.421052    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:52.603473 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:52.603422    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:53.179337 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:53.179293    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:53.429945 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:53.429827    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:54.293576 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:54.293535    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:55.123569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:55.123530    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:55.818125 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:55.818081    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:56.909068 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:56.909030    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:57.111874 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:57.111820    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:57.256807 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:57.256715    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:57.962602 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:57.962545    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:58.348273 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.348216    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:58.412631 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.412595    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"]
Nov 03 19:49:58.413306 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.413290    5237 cpu_manager.go:414] "containerMap: removing container" podUID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" containerName="istio-proxy"
Nov 03 19:49:58.413375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.413308    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" containerName="istio-proxy"
Nov 03 19:49:58.413375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.413319    5237 cpu_manager.go:414] "containerMap: removing container" podUID="e80c1008-2bd2-4a10-bacf-ec1d7148ebf8" containerName="payload-processing"
Nov 03 19:49:58.413375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.413325    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e80c1008-2bd2-4a10-bacf-ec1d7148ebf8" containerName="payload-processing"
Nov 03 19:49:58.413375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.413345    5237 cpu_manager.go:414] "containerMap: removing container" podUID="7f80a141-ae01-4ca9-911f-f2fe09d5492b" containerName="payload-pre-processing"
Nov 03 19:49:58.413375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.413350    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f80a141-ae01-4ca9-911f-f2fe09d5492b" containerName="payload-pre-processing"
Nov 03 19:49:58.413375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.413361    5237 cpu_manager.go:414] "containerMap: removing container" podUID="1c377e40-757e-47a6-b015-bf914f369081" containerName="istio-proxy"
Nov 03 19:49:58.413375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.413365    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c377e40-757e-47a6-b015-bf914f369081" containerName="istio-proxy"
Nov 03 19:49:58.413375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.413373    5237 cpu_manager.go:414] "containerMap: removing container" podUID="568c2367-6f5c-423f-9c5b-5cc760e4cb7b" containerName="main"
Nov 03 19:49:58.413375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.413377    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="568c2367-6f5c-423f-9c5b-5cc760e4cb7b" containerName="main"
Nov 03 19:49:58.413649 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.413497    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="33872d2d-bbf6-4242-a6c3-a0f3fffb54f7" containerName="istio-proxy"
Nov 03 19:49:58.413649 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.413505    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="e80c1008-2bd2-4a10-bacf-ec1d7148ebf8" containerName="payload-processing"
Nov 03 19:49:58.413649 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.413515    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f80a141-ae01-4ca9-911f-f2fe09d5492b" containerName="payload-pre-processing"
Nov 03 19:49:58.413649 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.413522    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c377e40-757e-47a6-b015-bf914f369081" containerName="istio-proxy"
Nov 03 19:49:58.413649 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.413527    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="568c2367-6f5c-423f-9c5b-5cc760e4cb7b" containerName="main"
Nov 03 19:49:58.418827 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.418798    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"
Nov 03 19:49:58.426810 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.426790    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-5444b0\""
Nov 03 19:49:58.448115 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.448073    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"]
Nov 03 19:49:58.458733 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.458696    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4d0eae8-11ce-483c-97f7-e0aef82ed54b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb\" (UID: \"a4d0eae8-11ce-483c-97f7-e0aef82ed54b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"
Nov 03 19:49:58.459059 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.459014    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwqx9\" (UniqueName: \"kubernetes.io/projected/a4d0eae8-11ce-483c-97f7-e0aef82ed54b-kube-api-access-gwqx9\") pod \"payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb\" (UID: \"a4d0eae8-11ce-483c-97f7-e0aef82ed54b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"
Nov 03 19:49:58.469184 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.469153    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"]
Nov 03 19:49:58.475009 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.474987    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"
Nov 03 19:49:58.477197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.476740    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-5444b0-dockercfg-8lz87\""
Nov 03 19:49:58.490431 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.490200    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"]
Nov 03 19:49:58.509066 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.509006    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:58.595862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.566672    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwqx9\" (UniqueName: \"kubernetes.io/projected/a4d0eae8-11ce-483c-97f7-e0aef82ed54b-kube-api-access-gwqx9\") pod \"payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb\" (UID: \"a4d0eae8-11ce-483c-97f7-e0aef82ed54b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"
Nov 03 19:49:58.595862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.566861    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4d0eae8-11ce-483c-97f7-e0aef82ed54b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb\" (UID: \"a4d0eae8-11ce-483c-97f7-e0aef82ed54b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"
Nov 03 19:49:58.595862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.567922    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4d0eae8-11ce-483c-97f7-e0aef82ed54b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb\" (UID: \"a4d0eae8-11ce-483c-97f7-e0aef82ed54b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"
Nov 03 19:49:58.723949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.693871    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27rjx\" (UniqueName: \"kubernetes.io/projected/e0e9e459-f64b-44b6-a217-56d4e11feada-kube-api-access-27rjx\") pod \"payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv\" (UID: \"e0e9e459-f64b-44b6-a217-56d4e11feada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"
Nov 03 19:49:58.723949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.694330    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e9e459-f64b-44b6-a217-56d4e11feada-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv\" (UID: \"e0e9e459-f64b-44b6-a217-56d4e11feada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"
Nov 03 19:49:58.777861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.767429    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:58.804094 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.791809    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwqx9\" (UniqueName: \"kubernetes.io/projected/a4d0eae8-11ce-483c-97f7-e0aef82ed54b-kube-api-access-gwqx9\") pod \"payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb\" (UID: \"a4d0eae8-11ce-483c-97f7-e0aef82ed54b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"
Nov 03 19:49:58.823850 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.820541    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e9e459-f64b-44b6-a217-56d4e11feada-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv\" (UID: \"e0e9e459-f64b-44b6-a217-56d4e11feada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"
Nov 03 19:49:58.823850 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.820647    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rjx\" (UniqueName: \"kubernetes.io/projected/e0e9e459-f64b-44b6-a217-56d4e11feada-kube-api-access-27rjx\") pod \"payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv\" (UID: \"e0e9e459-f64b-44b6-a217-56d4e11feada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"
Nov 03 19:49:58.865860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.865472    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e9e459-f64b-44b6-a217-56d4e11feada-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv\" (UID: \"e0e9e459-f64b-44b6-a217-56d4e11feada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"
Nov 03 19:49:58.916860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:58.915634    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27rjx\" (UniqueName: \"kubernetes.io/projected/e0e9e459-f64b-44b6-a217-56d4e11feada-kube-api-access-27rjx\") pod \"payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv\" (UID: \"e0e9e459-f64b-44b6-a217-56d4e11feada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"
Nov 03 19:49:59.034911 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:59.034021    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"
Nov 03 19:49:59.137888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:59.127306    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"
Nov 03 19:49:59.151559 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:59.148244    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:59.855983 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:49:59.855794    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d0eae8_11ce_483c_97f7_e0aef82ed54b.slice/crio-e50c9aac46baad21ba8873c28d2c9f22a70643b391e2f2cc7e4576f15f79405e WatchSource:0}: Error finding container e50c9aac46baad21ba8873c28d2c9f22a70643b391e2f2cc7e4576f15f79405e: Status 404 returned error can't find the container with id e50c9aac46baad21ba8873c28d2c9f22a70643b391e2f2cc7e4576f15f79405e
Nov 03 19:49:59.865653 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:59.865384    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"]
Nov 03 19:49:59.994256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:49:59.994225    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"]
Nov 03 19:49:59.996334 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:49:59.996306    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e9e459_f64b_44b6_a217_56d4e11feada.slice/crio-9d031f34137d1af69f826294ba5816eef5531bbc26118e8981827bc01db7d29e WatchSource:0}: Error finding container 9d031f34137d1af69f826294ba5816eef5531bbc26118e8981827bc01db7d29e: Status 404 returned error can't find the container with id 9d031f34137d1af69f826294ba5816eef5531bbc26118e8981827bc01db7d29e
Nov 03 19:50:00.035396 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:00.035361    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:00.197601 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:00.197555    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv" event={"ID":"e0e9e459-f64b-44b6-a217-56d4e11feada","Type":"ContainerStarted","Data":"473de8872a719b7f9acef52866d9867c2e0e6340656ce348f1674a4db1dd1563"}
Nov 03 19:50:00.197799 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:00.197636    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"
Nov 03 19:50:00.197799 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:00.197659    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv" event={"ID":"e0e9e459-f64b-44b6-a217-56d4e11feada","Type":"ContainerStarted","Data":"9d031f34137d1af69f826294ba5816eef5531bbc26118e8981827bc01db7d29e"}
Nov 03 19:50:00.199645 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:00.199604    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb" event={"ID":"a4d0eae8-11ce-483c-97f7-e0aef82ed54b","Type":"ContainerStarted","Data":"1c9d739eb58e78a7ec8d8f9431ba30a1beea932f7ffb1068a924029b1c58c6ff"}
Nov 03 19:50:00.199782 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:00.199674    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"
Nov 03 19:50:00.199782 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:00.199692    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb" event={"ID":"a4d0eae8-11ce-483c-97f7-e0aef82ed54b","Type":"ContainerStarted","Data":"e50c9aac46baad21ba8873c28d2c9f22a70643b391e2f2cc7e4576f15f79405e"}
Nov 03 19:50:00.220050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:00.219997    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv" podStartSLOduration=2.219978145 podStartE2EDuration="2.219978145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:00.217986524 +0000 UTC m=+1356.472293749" watchObservedRunningTime="2026-11-03 19:50:00.219978145 +0000 UTC m=+1356.474285370"
Nov 03 19:50:00.236904 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:00.236855    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb" podStartSLOduration=2.236829698 podStartE2EDuration="2.236829698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:00.235719808 +0000 UTC m=+1356.490027033" watchObservedRunningTime="2026-11-03 19:50:00.236829698 +0000 UTC m=+1356.491136922"
Nov 03 19:50:00.249021 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:00.248991    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:00.718859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:00.714240    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:01.128435 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:01.128398    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:01.989994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:01.989958    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:02.666156 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:02.666121    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:02.798578 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:02.798520    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:03.702779 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:03.702741    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:04.626857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:04.618147    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:05.054853 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:05.054805    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:05.870413 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:05.870370    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:06.109861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:06.089236    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:07.148869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:07.148821    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:07.620011 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:07.619969    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"]
Nov 03 19:50:07.765420 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:07.765376    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:07.994921 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:07.994487    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:08.238603 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:08.238563    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:08.417971 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:08.412338    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:08.667594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:08.667553    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:09.555065 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:09.555023    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:09.753223 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:09.753188    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:09.902899 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:09.902787    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:10.475858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:10.475428    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:10.639856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:10.639114    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:10.992078 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:10.992024    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:11.191716 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:11.191679    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:11.258274 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:11.258194    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"
Nov 03 19:50:11.315103 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:11.315071    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"
Nov 03 19:50:12.171426 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:12.171381    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:12.309010 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:12.308970    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:12.440862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:12.440740    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:13.124011 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:13.123977    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:13.259391 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:50:13.259350    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:50:13.261463 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:13.261421    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:14.512905 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:14.512862    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:15.414500 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:15.414470    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:16.132962 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:16.132925    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:16.265764 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:16.265726    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:16.420960 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:16.420811    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:16.574737 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:16.571969    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:17.512669 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:17.512625    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:17.718894 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:17.709268    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:18.732455 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:18.732417    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:19.534556 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:19.534519    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:20.154657 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:20.142996    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:21.228125 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:21.228087    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:22.037857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:22.037802    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:22.665369 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:22.665326    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main" containerID="cri-o://a5293711efcc818e407dd8706050ed530915acf3e02106eb393fbb12bf47b021" gracePeriod=45
Nov 03 19:50:22.693948 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:22.693910    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:22.884150 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:22.884015    5237 generic.go:356] "Generic (PLEG): container finished" podID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerID="a5293711efcc818e407dd8706050ed530915acf3e02106eb393fbb12bf47b021" exitCode=0
Nov 03 19:50:22.884150 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:22.884094    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" event={"ID":"04d00e3f-7b7f-4bde-a443-77aadc84642b","Type":"ContainerDied","Data":"a5293711efcc818e407dd8706050ed530915acf3e02106eb393fbb12bf47b021"}
Nov 03 19:50:23.018791 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.018756    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:50:23.132067 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.132027    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-tmp-dir\") pod \"04d00e3f-7b7f-4bde-a443-77aadc84642b\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") "
Nov 03 19:50:23.132272 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.132146    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d00e3f-7b7f-4bde-a443-77aadc84642b-kube-api-access-zj2bc\" (UniqueName: \"kubernetes.io/projected/04d00e3f-7b7f-4bde-a443-77aadc84642b-kube-api-access-zj2bc\") pod \"04d00e3f-7b7f-4bde-a443-77aadc84642b\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") "
Nov 03 19:50:23.132272 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.132173    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-model-cache\") pod \"04d00e3f-7b7f-4bde-a443-77aadc84642b\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") "
Nov 03 19:50:23.132272 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.132219    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04d00e3f-7b7f-4bde-a443-77aadc84642b-tls-certs\" (UniqueName: \"kubernetes.io/secret/04d00e3f-7b7f-4bde-a443-77aadc84642b-tls-certs\") pod \"04d00e3f-7b7f-4bde-a443-77aadc84642b\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") "
Nov 03 19:50:23.132445 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.132373    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-tmp-dir" pod "04d00e3f-7b7f-4bde-a443-77aadc84642b" (UID: "04d00e3f-7b7f-4bde-a443-77aadc84642b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:23.132508 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.132467    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-dshm\") pod \"04d00e3f-7b7f-4bde-a443-77aadc84642b\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") "
Nov 03 19:50:23.132569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.132547    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-home\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-home\") pod \"04d00e3f-7b7f-4bde-a443-77aadc84642b\" (UID: \"04d00e3f-7b7f-4bde-a443-77aadc84642b\") "
Nov 03 19:50:23.132876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.132815    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-home" pod "04d00e3f-7b7f-4bde-a443-77aadc84642b" (UID: "04d00e3f-7b7f-4bde-a443-77aadc84642b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:23.132986 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.132957    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-model-cache" pod "04d00e3f-7b7f-4bde-a443-77aadc84642b" (UID: "04d00e3f-7b7f-4bde-a443-77aadc84642b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:23.133369 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.133343    5237 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-home\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:23.133468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.133373    5237 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-tmp-dir\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:23.133468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.133386    5237 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-model-cache\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:23.135386 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.135360    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-dshm" pod "04d00e3f-7b7f-4bde-a443-77aadc84642b" (UID: "04d00e3f-7b7f-4bde-a443-77aadc84642b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:23.135386 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.135362    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04d00e3f-7b7f-4bde-a443-77aadc84642b-tls-certs" pod "04d00e3f-7b7f-4bde-a443-77aadc84642b" (UID: "04d00e3f-7b7f-4bde-a443-77aadc84642b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:50:23.137031 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.137007    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d00e3f-7b7f-4bde-a443-77aadc84642b-kube-api-access-zj2bc" pod "04d00e3f-7b7f-4bde-a443-77aadc84642b" (UID: "04d00e3f-7b7f-4bde-a443-77aadc84642b"). InnerVolumeSpecName "kube-api-access-zj2bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:50:23.235920 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.235776    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj2bc\" (UniqueName: \"kubernetes.io/projected/04d00e3f-7b7f-4bde-a443-77aadc84642b-kube-api-access-zj2bc\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:23.235920 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.235819    5237 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04d00e3f-7b7f-4bde-a443-77aadc84642b-tls-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:23.235920 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.235850    5237 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04d00e3f-7b7f-4bde-a443-77aadc84642b-dshm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:23.391286 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.391248    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:23.894312 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.894288    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"
Nov 03 19:50:23.894749 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.894333    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv" event={"ID":"04d00e3f-7b7f-4bde-a443-77aadc84642b","Type":"ContainerDied","Data":"f78ba6e360af28f53fe4863edfb57c35a252a6cbb57a8b7608b068f11e80096c"}
Nov 03 19:50:23.894749 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.894398    5237 scope.go:122] "RemoveContainer" containerID="a5293711efcc818e407dd8706050ed530915acf3e02106eb393fbb12bf47b021"
Nov 03 19:50:23.916454 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.916419    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"]
Nov 03 19:50:23.922145 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.922122    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/distinct-2-w6-c5f6cb-kserve-66f478598f-cr9dv"]
Nov 03 19:50:23.987530 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:23.987497    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"]
Nov 03 19:50:24.091126 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:24.091091    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:24.237309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:24.237223    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:24.380864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:24.379822    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:25.141794 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:25.141619    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:25.715384 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:25.715351    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:25.862353 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:25.862324    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" path="/var/lib/kubelet/pods/04d00e3f-7b7f-4bde-a443-77aadc84642b/volumes"
Nov 03 19:50:26.636458 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:26.636423    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:26.789560 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:26.782144    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:26.975553 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:26.975449    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:27.634217 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:27.634182    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:28.488392 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:28.488353    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:28.987863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:28.987792    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr" podUID="cf3379f5-4f44-43e1-beb2-9ea9b3511be1" containerName="payload-pre-processing" containerID="cri-o://d6c58eed3fa08f80bb1f5ee4905b2211736461d33815b3ed231da688b58149ce" gracePeriod=25
Nov 03 19:50:29.144731 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:29.144686    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:29.412150 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:29.412117    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"
Nov 03 19:50:29.511499 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:29.511466    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf3379f5-4f44-43e1-beb2-9ea9b3511be1-kube-api-access-dd9xd\" (UniqueName: \"kubernetes.io/projected/cf3379f5-4f44-43e1-beb2-9ea9b3511be1-kube-api-access-dd9xd\") pod \"cf3379f5-4f44-43e1-beb2-9ea9b3511be1\" (UID: \"cf3379f5-4f44-43e1-beb2-9ea9b3511be1\") "
Nov 03 19:50:29.512021 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:29.511570    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf3379f5-4f44-43e1-beb2-9ea9b3511be1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf3379f5-4f44-43e1-beb2-9ea9b3511be1-plugins-config-volume\") pod \"cf3379f5-4f44-43e1-beb2-9ea9b3511be1\" (UID: \"cf3379f5-4f44-43e1-beb2-9ea9b3511be1\") "
Nov 03 19:50:29.512021 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:29.511968    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3379f5-4f44-43e1-beb2-9ea9b3511be1-plugins-config-volume" pod "cf3379f5-4f44-43e1-beb2-9ea9b3511be1" (UID: "cf3379f5-4f44-43e1-beb2-9ea9b3511be1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:50:29.512468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:29.512444    5237 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf3379f5-4f44-43e1-beb2-9ea9b3511be1-plugins-config-volume\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:29.514310 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:29.514277    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3379f5-4f44-43e1-beb2-9ea9b3511be1-kube-api-access-dd9xd" pod "cf3379f5-4f44-43e1-beb2-9ea9b3511be1" (UID: "cf3379f5-4f44-43e1-beb2-9ea9b3511be1"). InnerVolumeSpecName "kube-api-access-dd9xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:50:29.613822 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:29.613741    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd9xd\" (UniqueName: \"kubernetes.io/projected/cf3379f5-4f44-43e1-beb2-9ea9b3511be1-kube-api-access-dd9xd\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:29.850489 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:29.850447    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:30.010444 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:30.010407    5237 generic.go:356] "Generic (PLEG): container finished" podID="cf3379f5-4f44-43e1-beb2-9ea9b3511be1" containerID="d6c58eed3fa08f80bb1f5ee4905b2211736461d33815b3ed231da688b58149ce" exitCode=0
Nov 03 19:50:30.010617 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:30.010470    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"
Nov 03 19:50:30.010617 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:30.010499    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr" event={"ID":"cf3379f5-4f44-43e1-beb2-9ea9b3511be1","Type":"ContainerDied","Data":"d6c58eed3fa08f80bb1f5ee4905b2211736461d33815b3ed231da688b58149ce"}
Nov 03 19:50:30.010617 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:30.010577    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr" event={"ID":"cf3379f5-4f44-43e1-beb2-9ea9b3511be1","Type":"ContainerDied","Data":"657986b624ea9c406a91cabfea19f9927c9c696a7cd87869a464a28f9b11c5dc"}
Nov 03 19:50:30.010617 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:30.010602    5237 scope.go:122] "RemoveContainer" containerID="d6c58eed3fa08f80bb1f5ee4905b2211736461d33815b3ed231da688b58149ce"
Nov 03 19:50:30.020962 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:30.020939    5237 scope.go:122] "RemoveContainer" containerID="d6c58eed3fa08f80bb1f5ee4905b2211736461d33815b3ed231da688b58149ce"
Nov 03 19:50:30.021248 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:50:30.021228    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c58eed3fa08f80bb1f5ee4905b2211736461d33815b3ed231da688b58149ce\": container with ID starting with d6c58eed3fa08f80bb1f5ee4905b2211736461d33815b3ed231da688b58149ce not found: ID does not exist" containerID="d6c58eed3fa08f80bb1f5ee4905b2211736461d33815b3ed231da688b58149ce"
Nov 03 19:50:30.021314 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:30.021260    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c58eed3fa08f80bb1f5ee4905b2211736461d33815b3ed231da688b58149ce"} err="failed to get container status \"d6c58eed3fa08f80bb1f5ee4905b2211736461d33815b3ed231da688b58149ce\": rpc error: code = NotFound desc = could not find container \"d6c58eed3fa08f80bb1f5ee4905b2211736461d33815b3ed231da688b58149ce\": container with ID starting with d6c58eed3fa08f80bb1f5ee4905b2211736461d33815b3ed231da688b58149ce not found: ID does not exist"
Nov 03 19:50:30.033305 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:30.033279    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"]
Nov 03 19:50:30.036709 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:30.036684    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10e551-67b5d87c5-w88rr"]
Nov 03 19:50:30.566506 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:30.563565    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:30.748065 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:30.748030    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:31.327380 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.327341    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:31.491678 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.491373    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:31.529632 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.529600    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"]
Nov 03 19:50:31.531037 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.531011    5237 cpu_manager.go:414] "containerMap: removing container" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main"
Nov 03 19:50:31.531209 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.531193    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main"
Nov 03 19:50:31.531359 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.531346    5237 cpu_manager.go:414] "containerMap: removing container" podUID="cf3379f5-4f44-43e1-beb2-9ea9b3511be1" containerName="payload-pre-processing"
Nov 03 19:50:31.531462 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.531448    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf3379f5-4f44-43e1-beb2-9ea9b3511be1" containerName="payload-pre-processing"
Nov 03 19:50:31.531794 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.531777    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d00e3f-7b7f-4bde-a443-77aadc84642b" containerName="main"
Nov 03 19:50:31.531966 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.531951    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf3379f5-4f44-43e1-beb2-9ea9b3511be1" containerName="payload-pre-processing"
Nov 03 19:50:31.539765 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.539744    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"]
Nov 03 19:50:31.539889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.539849    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.617790 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.617714    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-477cd8-openshift-default-dockercfg-vk2t9\""
Nov 03 19:50:31.624662 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.624625    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-workload-certs\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.624849 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.624715    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-workload-socket\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.624937 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.624846    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqnj6\" (UniqueName: \"kubernetes.io/projected/6037ccaa-48d9-450c-8726-213ae6b76500-kube-api-access-lqnj6\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.625004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.624973    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6037ccaa-48d9-450c-8726-213ae6b76500-istiod-ca-cert\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.625146 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.625118    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-istio-data\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.625279 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.625199    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6037ccaa-48d9-450c-8726-213ae6b76500-istio-token\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.625350 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.625295    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6037ccaa-48d9-450c-8726-213ae6b76500-istio-podinfo\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.625472 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.625377    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-credential-socket\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.625472 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.625402    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-istio-envoy\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.625592 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.625517    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:31.727439 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.727402    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-istio-data\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.727439 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.727439    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6037ccaa-48d9-450c-8726-213ae6b76500-istio-token\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.727696 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.727488    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6037ccaa-48d9-450c-8726-213ae6b76500-istio-podinfo\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.727696 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.727542    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-credential-socket\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.727696 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.727570    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-istio-envoy\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.727696 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.727660    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-workload-certs\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.727696 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.727693    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-workload-socket\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.728022 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.727749    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqnj6\" (UniqueName: \"kubernetes.io/projected/6037ccaa-48d9-450c-8726-213ae6b76500-kube-api-access-lqnj6\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.728022 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.727849    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6037ccaa-48d9-450c-8726-213ae6b76500-istiod-ca-cert\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.728022 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.727895    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-istio-data\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.728022 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.727931    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-credential-socket\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.728234 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.728187    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-workload-socket\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.728234 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.728219    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-workload-certs\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.728523 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.728501    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6037ccaa-48d9-450c-8726-213ae6b76500-istiod-ca-cert\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.730053 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.730029    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-istio-envoy\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.730176 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.730161    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6037ccaa-48d9-450c-8726-213ae6b76500-istio-podinfo\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.753168 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.753140    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6037ccaa-48d9-450c-8726-213ae6b76500-istio-token\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.753293 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.753181    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqnj6\" (UniqueName: \"kubernetes.io/projected/6037ccaa-48d9-450c-8726-213ae6b76500-kube-api-access-lqnj6\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.856068 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.855821    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:31.865608 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:31.865571    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf3379f5-4f44-43e1-beb2-9ea9b3511be1" path="/var/lib/kubelet/pods/cf3379f5-4f44-43e1-beb2-9ea9b3511be1/volumes"
Nov 03 19:50:32.358211 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:32.358181    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"]
Nov 03 19:50:32.359021 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:50:32.358984    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6037ccaa_48d9_450c_8726_213ae6b76500.slice/crio-5fa2d0e247ea8eda73c78c92440947eb356ad26fe286ee15373f3de3b4703a4d WatchSource:0}: Error finding container 5fa2d0e247ea8eda73c78c92440947eb356ad26fe286ee15373f3de3b4703a4d: Status 404 returned error can't find the container with id 5fa2d0e247ea8eda73c78c92440947eb356ad26fe286ee15373f3de3b4703a4d
Nov 03 19:50:32.361897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:32.361850    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:50:32.362002 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:32.361946    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:50:32.362002 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:32.361995    5237 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:50:32.468858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:32.468075    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:33.246182 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:33.246140    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6" event={"ID":"6037ccaa-48d9-450c-8726-213ae6b76500","Type":"ContainerStarted","Data":"b1acecc213f06a95d54e18185306498c7f752b93a6d4ed10dd5d33b30e2fcabe"}
Nov 03 19:50:33.246622 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:33.246199    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6" event={"ID":"6037ccaa-48d9-450c-8726-213ae6b76500","Type":"ContainerStarted","Data":"5fa2d0e247ea8eda73c78c92440947eb356ad26fe286ee15373f3de3b4703a4d"}
Nov 03 19:50:33.346508 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:33.337542    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6" podStartSLOduration=2.337517534 podStartE2EDuration="2.337517534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:33.289033028 +0000 UTC m=+1389.543340281" watchObservedRunningTime="2026-11-03 19:50:33.337517534 +0000 UTC m=+1389.591824759"
Nov 03 19:50:33.369861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:33.359218    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:33.936861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:33.935981    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:50:33.936861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:33.936044    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6" podUID="6037ccaa-48d9-450c-8726-213ae6b76500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:50:34.064869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:34.057771    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:34.361136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:34.361098    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:34.856828 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:34.856797    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:50:34.857045 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:34.856876    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6" podUID="6037ccaa-48d9-450c-8726-213ae6b76500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:50:35.213597 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:35.213498    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:35.856326 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:35.856290    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:50:35.856821 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:35.856355    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6" podUID="6037ccaa-48d9-450c-8726-213ae6b76500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:50:35.987863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:35.982636    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:36.197324 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:36.197232    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:36.745192 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:36.745158    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:36.857190 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:36.857138    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:50:36.857860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:36.857812    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6" podUID="6037ccaa-48d9-450c-8726-213ae6b76500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:50:37.620190 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:37.620157    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:37.856372 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:37.856337    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:50:37.856574 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:37.856382    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6" podUID="6037ccaa-48d9-450c-8726-213ae6b76500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:50:37.894450 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:37.894288    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:38.629268 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:38.629222    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:38.856634 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:38.856589    5237 patch_prober.go:29] interesting pod/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:50:38.856847 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:38.856667    5237 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6" podUID="6037ccaa-48d9-450c-8726-213ae6b76500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:50:39.863041 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:39.863014    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:39.863469 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:39.863453    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:39.864161 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:39.864142    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:50:41.288992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:41.288960    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:41.438904 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:41.438871    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:42.283418 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:42.283374    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:42.847382 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:42.847334    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:43.165543 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:43.165453    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:43.336072 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:50:43.336011    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:50:43.542807 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:43.542772    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:43.689857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:43.687373    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:44.108327 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:44.108286    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:44.687988 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:44.687951    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:45.344863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:45.340397    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:45.538741 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:45.538707    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:45.688858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:45.686901    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"]
Nov 03 19:50:45.707046 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:45.706998    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"]
Nov 03 19:50:45.711153 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:45.710329    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:45.850387 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:45.850352    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:46.023689 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:46.023589    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:46.217704 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:46.217669    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:46.624108 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:46.624072    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:46.938947 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:46.929248    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:47.897922 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:47.894962    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:48.318694 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:48.317539    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:49.074857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:49.073562    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:49.224083 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:49.223991    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:49.380905 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:49.376879    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:50.027544 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:50.027447    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:50.170336 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:50.170301    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:51.076883 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:51.060098    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:51.312857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:51.309930    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:51.512913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:51.512582    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:51.921852 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:51.916645    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:52.071501 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:52.070948    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:53.527306 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:53.527262    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:53.686924 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:53.686884    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:54.146236 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:54.146196    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:54.165868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:54.165554    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"]
Nov 03 19:50:54.166060 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:54.165923    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk" podUID="0398a799-4e11-4f8e-913f-8a42574c0ae9" containerName="istio-proxy" containerID="cri-o://d72f758659f5ff0a21e09c6338854b7067c9ba2c5a4aeb20101eb6cffeb53402" gracePeriod=30
Nov 03 19:50:54.227985 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:54.227951    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"]
Nov 03 19:50:54.228573 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:54.228528    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn" podUID="e56d7264-7d35-4dd8-946b-c1ab4d40e941" containerName="istio-proxy" containerID="cri-o://ede69654eeddbd2f71333aacdeaeac2655cd04ffe0046ba709d74a004459b88f" gracePeriod=30
Nov 03 19:50:54.301090 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:54.301057    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"]
Nov 03 19:50:54.301581 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:54.301547    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" podUID="22e9f189-e173-4b40-b4a2-3455be004378" containerName="istio-proxy" containerID="cri-o://c932bcc9628f0f0e01ee8e29f48e5d96f4d00e307ac8d0cca6023982b936e188" gracePeriod=30
Nov 03 19:50:55.282164 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:55.282118    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:55.292412 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:55.291689    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:55.334877 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:55.324961    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:55.459870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:55.456077    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:55.459870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:55.456435    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:55.477885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:55.475422    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:55.477885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:55.477102    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:55.493874 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:55.480895    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 19:50:55.531029 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:55.530987    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:55.709387 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:55.709350    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:55.912341 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:55.912307    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:56.241868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:56.235941    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-f5xfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:50:56.241868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:56.236008    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb" podUID="98ec8c1c-49db-492e-958e-fabe09d044fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:56.890524 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:56.890402    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:57.248428 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:57.241063    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:58.061597 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:58.061555    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:58.391513 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:58.391403    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:58.549050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:58.549012    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:59.103026 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.102975    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:59.587721 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.587687    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:50:59.817955 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.792446    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:59.817955 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.793067    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-data\") pod \"0398a799-4e11-4f8e-913f-8a42574c0ae9\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") "
Nov 03 19:50:59.817955 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.793118    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-credential-socket\") pod \"0398a799-4e11-4f8e-913f-8a42574c0ae9\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") "
Nov 03 19:50:59.817955 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.793225    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-workload-certs\") pod \"0398a799-4e11-4f8e-913f-8a42574c0ae9\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") "
Nov 03 19:50:59.817955 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.793286    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-podinfo\") pod \"0398a799-4e11-4f8e-913f-8a42574c0ae9\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") "
Nov 03 19:50:59.817955 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.793313    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-token\" (UniqueName: \"kubernetes.io/projected/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-token\") pod \"0398a799-4e11-4f8e-913f-8a42574c0ae9\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") "
Nov 03 19:50:59.817955 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.793344    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0398a799-4e11-4f8e-913f-8a42574c0ae9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0398a799-4e11-4f8e-913f-8a42574c0ae9-istiod-ca-cert\") pod \"0398a799-4e11-4f8e-913f-8a42574c0ae9\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") "
Nov 03 19:50:59.817955 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.793457    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0398a799-4e11-4f8e-913f-8a42574c0ae9-kube-api-access-dn4fs\" (UniqueName: \"kubernetes.io/projected/0398a799-4e11-4f8e-913f-8a42574c0ae9-kube-api-access-dn4fs\") pod \"0398a799-4e11-4f8e-913f-8a42574c0ae9\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") "
Nov 03 19:50:59.817955 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.793522    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-workload-socket\") pod \"0398a799-4e11-4f8e-913f-8a42574c0ae9\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") "
Nov 03 19:50:59.817955 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.793721    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-envoy\") pod \"0398a799-4e11-4f8e-913f-8a42574c0ae9\" (UID: \"0398a799-4e11-4f8e-913f-8a42574c0ae9\") "
Nov 03 19:50:59.828885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.821984    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-data" pod "0398a799-4e11-4f8e-913f-8a42574c0ae9" (UID: "0398a799-4e11-4f8e-913f-8a42574c0ae9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:59.828885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.822185    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-workload-certs" pod "0398a799-4e11-4f8e-913f-8a42574c0ae9" (UID: "0398a799-4e11-4f8e-913f-8a42574c0ae9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:59.828885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.822606    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0398a799-4e11-4f8e-913f-8a42574c0ae9-istiod-ca-cert" pod "0398a799-4e11-4f8e-913f-8a42574c0ae9" (UID: "0398a799-4e11-4f8e-913f-8a42574c0ae9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:50:59.828885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.822934    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-workload-socket" pod "0398a799-4e11-4f8e-913f-8a42574c0ae9" (UID: "0398a799-4e11-4f8e-913f-8a42574c0ae9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:59.828885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.824349    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-podinfo" pod "0398a799-4e11-4f8e-913f-8a42574c0ae9" (UID: "0398a799-4e11-4f8e-913f-8a42574c0ae9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:50:59.829305 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.828898    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-credential-socket" pod "0398a799-4e11-4f8e-913f-8a42574c0ae9" (UID: "0398a799-4e11-4f8e-913f-8a42574c0ae9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:59.860872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.858378    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:59.860872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.858645    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:59.860872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.859030    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:59.860872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.859059    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:59.860872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.859078    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0398a799-4e11-4f8e-913f-8a42574c0ae9-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:59.860872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.859096    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:59.879385 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.878728    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0398a799-4e11-4f8e-913f-8a42574c0ae9-kube-api-access-dn4fs" pod "0398a799-4e11-4f8e-913f-8a42574c0ae9" (UID: "0398a799-4e11-4f8e-913f-8a42574c0ae9"). InnerVolumeSpecName "kube-api-access-dn4fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:50:59.879871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.879820    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-envoy" pod "0398a799-4e11-4f8e-913f-8a42574c0ae9" (UID: "0398a799-4e11-4f8e-913f-8a42574c0ae9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:59.880688 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.880288    5237 generic.go:356] "Generic (PLEG): container finished" podID="22e9f189-e173-4b40-b4a2-3455be004378" containerID="c932bcc9628f0f0e01ee8e29f48e5d96f4d00e307ac8d0cca6023982b936e188" exitCode=0
Nov 03 19:50:59.882065 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.882011    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-token" pod "0398a799-4e11-4f8e-913f-8a42574c0ae9" (UID: "0398a799-4e11-4f8e-913f-8a42574c0ae9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:50:59.885818 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.885794    5237 generic.go:356] "Generic (PLEG): container finished" podID="0398a799-4e11-4f8e-913f-8a42574c0ae9" containerID="d72f758659f5ff0a21e09c6338854b7067c9ba2c5a4aeb20101eb6cffeb53402" exitCode=0
Nov 03 19:50:59.886125 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.886106    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"
Nov 03 19:50:59.906429 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.906323    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" event={"ID":"22e9f189-e173-4b40-b4a2-3455be004378","Type":"ContainerDied","Data":"c932bcc9628f0f0e01ee8e29f48e5d96f4d00e307ac8d0cca6023982b936e188"}
Nov 03 19:50:59.906429 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.906415    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk" event={"ID":"0398a799-4e11-4f8e-913f-8a42574c0ae9","Type":"ContainerDied","Data":"d72f758659f5ff0a21e09c6338854b7067c9ba2c5a4aeb20101eb6cffeb53402"}
Nov 03 19:50:59.906663 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.906470    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk" event={"ID":"0398a799-4e11-4f8e-913f-8a42574c0ae9","Type":"ContainerDied","Data":"7efdda866eb53c22200cf6efb3343297eecbaafb050fbcdf71c5662cec0d12a5"}
Nov 03 19:50:59.906663 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.906508    5237 scope.go:122] "RemoveContainer" containerID="d72f758659f5ff0a21e09c6338854b7067c9ba2c5a4aeb20101eb6cffeb53402"
Nov 03 19:50:59.919985 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.919696    5237 generic.go:356] "Generic (PLEG): container finished" podID="e56d7264-7d35-4dd8-946b-c1ab4d40e941" containerID="ede69654eeddbd2f71333aacdeaeac2655cd04ffe0046ba709d74a004459b88f" exitCode=0
Nov 03 19:50:59.919985 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.919771    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn" event={"ID":"e56d7264-7d35-4dd8-946b-c1ab4d40e941","Type":"ContainerDied","Data":"ede69654eeddbd2f71333aacdeaeac2655cd04ffe0046ba709d74a004459b88f"}
Nov 03 19:50:59.934326 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.934299    5237 scope.go:122] "RemoveContainer" containerID="d72f758659f5ff0a21e09c6338854b7067c9ba2c5a4aeb20101eb6cffeb53402"
Nov 03 19:50:59.934607 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:50:59.934587    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d72f758659f5ff0a21e09c6338854b7067c9ba2c5a4aeb20101eb6cffeb53402\": container with ID starting with d72f758659f5ff0a21e09c6338854b7067c9ba2c5a4aeb20101eb6cffeb53402 not found: ID does not exist" containerID="d72f758659f5ff0a21e09c6338854b7067c9ba2c5a4aeb20101eb6cffeb53402"
Nov 03 19:50:59.934698 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.934617    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d72f758659f5ff0a21e09c6338854b7067c9ba2c5a4aeb20101eb6cffeb53402"} err="failed to get container status \"d72f758659f5ff0a21e09c6338854b7067c9ba2c5a4aeb20101eb6cffeb53402\": rpc error: code = NotFound desc = could not find container \"d72f758659f5ff0a21e09c6338854b7067c9ba2c5a4aeb20101eb6cffeb53402\": container with ID starting with d72f758659f5ff0a21e09c6338854b7067c9ba2c5a4aeb20101eb6cffeb53402 not found: ID does not exist"
Nov 03 19:50:59.938092 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.938069    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:50:59.947857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.946588    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"]
Nov 03 19:50:59.953336 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.953305    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-vgqbk"]
Nov 03 19:50:59.962148 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.962115    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:59.962277 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.962167    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn4fs\" (UniqueName: \"kubernetes.io/projected/0398a799-4e11-4f8e-913f-8a42574c0ae9-kube-api-access-dn4fs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:59.962277 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:50:59.962183    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0398a799-4e11-4f8e-913f-8a42574c0ae9-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.029804 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.029768    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:51:00.063750 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.063710    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-credential-socket\") pod \"22e9f189-e173-4b40-b4a2-3455be004378\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") "
Nov 03 19:51:00.063960 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.063809    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-workload-socket\") pod \"22e9f189-e173-4b40-b4a2-3455be004378\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") "
Nov 03 19:51:00.063960 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.063901    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e9f189-e173-4b40-b4a2-3455be004378-kube-api-access-9cmfd\" (UniqueName: \"kubernetes.io/projected/22e9f189-e173-4b40-b4a2-3455be004378-kube-api-access-9cmfd\") pod \"22e9f189-e173-4b40-b4a2-3455be004378\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") "
Nov 03 19:51:00.064132 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.063974    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22e9f189-e173-4b40-b4a2-3455be004378-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e9f189-e173-4b40-b4a2-3455be004378-istiod-ca-cert\") pod \"22e9f189-e173-4b40-b4a2-3455be004378\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") "
Nov 03 19:51:00.064132 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.064019    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-istio-envoy\") pod \"22e9f189-e173-4b40-b4a2-3455be004378\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") "
Nov 03 19:51:00.064132 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.064047    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e9f189-e173-4b40-b4a2-3455be004378-istio-token\" (UniqueName: \"kubernetes.io/projected/22e9f189-e173-4b40-b4a2-3455be004378-istio-token\") pod \"22e9f189-e173-4b40-b4a2-3455be004378\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") "
Nov 03 19:51:00.064132 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.064080    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-istio-data\") pod \"22e9f189-e173-4b40-b4a2-3455be004378\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") "
Nov 03 19:51:00.064360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.064212    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-workload-certs\") pod \"22e9f189-e173-4b40-b4a2-3455be004378\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") "
Nov 03 19:51:00.064360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.064250    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22e9f189-e173-4b40-b4a2-3455be004378-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e9f189-e173-4b40-b4a2-3455be004378-istio-podinfo\") pod \"22e9f189-e173-4b40-b4a2-3455be004378\" (UID: \"22e9f189-e173-4b40-b4a2-3455be004378\") "
Nov 03 19:51:00.065676 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.065643    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e9f189-e173-4b40-b4a2-3455be004378-istiod-ca-cert" pod "22e9f189-e173-4b40-b4a2-3455be004378" (UID: "22e9f189-e173-4b40-b4a2-3455be004378"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:00.065808 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.065742    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-workload-socket" pod "22e9f189-e173-4b40-b4a2-3455be004378" (UID: "22e9f189-e173-4b40-b4a2-3455be004378"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:00.066013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.065988    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-credential-socket" pod "22e9f189-e173-4b40-b4a2-3455be004378" (UID: "22e9f189-e173-4b40-b4a2-3455be004378"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:00.066340 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.066280    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-workload-certs" pod "22e9f189-e173-4b40-b4a2-3455be004378" (UID: "22e9f189-e173-4b40-b4a2-3455be004378"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:00.067041 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.067016    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-istio-data" pod "22e9f189-e173-4b40-b4a2-3455be004378" (UID: "22e9f189-e173-4b40-b4a2-3455be004378"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:00.069210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.069181    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e9f189-e173-4b40-b4a2-3455be004378-istio-token" pod "22e9f189-e173-4b40-b4a2-3455be004378" (UID: "22e9f189-e173-4b40-b4a2-3455be004378"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:00.074288 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.074260    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e9f189-e173-4b40-b4a2-3455be004378-kube-api-access-9cmfd" pod "22e9f189-e173-4b40-b4a2-3455be004378" (UID: "22e9f189-e173-4b40-b4a2-3455be004378"). InnerVolumeSpecName "kube-api-access-9cmfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:00.074442 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.074421    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22e9f189-e173-4b40-b4a2-3455be004378-istio-podinfo" pod "22e9f189-e173-4b40-b4a2-3455be004378" (UID: "22e9f189-e173-4b40-b4a2-3455be004378"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:51:00.075677 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.075645    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-istio-envoy" pod "22e9f189-e173-4b40-b4a2-3455be004378" (UID: "22e9f189-e173-4b40-b4a2-3455be004378"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:00.165688 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.165608    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-envoy\") pod \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") "
Nov 03 19:51:00.166122 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.165697    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-workload-socket\") pod \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") "
Nov 03 19:51:00.166122 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.165770    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-podinfo\") pod \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") "
Nov 03 19:51:00.166122 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.165931    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56d7264-7d35-4dd8-946b-c1ab4d40e941-kube-api-access-98r7z\" (UniqueName: \"kubernetes.io/projected/e56d7264-7d35-4dd8-946b-c1ab4d40e941-kube-api-access-98r7z\") pod \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") "
Nov 03 19:51:00.166122 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.166000    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-credential-socket\") pod \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") "
Nov 03 19:51:00.166122 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.166035    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-workload-certs\") pod \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") "
Nov 03 19:51:00.166122 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.166045    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-workload-socket" pod "e56d7264-7d35-4dd8-946b-c1ab4d40e941" (UID: "e56d7264-7d35-4dd8-946b-c1ab4d40e941"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:00.166122 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.166066    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-token\" (UniqueName: \"kubernetes.io/projected/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-token\") pod \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") "
Nov 03 19:51:00.166526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.166140    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istiod-ca-cert\") pod \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") "
Nov 03 19:51:00.166526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.166288    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-credential-socket" pod "e56d7264-7d35-4dd8-946b-c1ab4d40e941" (UID: "e56d7264-7d35-4dd8-946b-c1ab4d40e941"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:00.166526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.166304    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-workload-certs" pod "e56d7264-7d35-4dd8-946b-c1ab4d40e941" (UID: "e56d7264-7d35-4dd8-946b-c1ab4d40e941"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:00.166526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.166314    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-data\") pod \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\" (UID: \"e56d7264-7d35-4dd8-946b-c1ab4d40e941\") "
Nov 03 19:51:00.166734 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.166566    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-data" pod "e56d7264-7d35-4dd8-946b-c1ab4d40e941" (UID: "e56d7264-7d35-4dd8-946b-c1ab4d40e941"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:00.166734 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.166673    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istiod-ca-cert" pod "e56d7264-7d35-4dd8-946b-c1ab4d40e941" (UID: "e56d7264-7d35-4dd8-946b-c1ab4d40e941"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:00.167284 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.167257    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.167416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.167287    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.167416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.167301    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.167416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.167317    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e9f189-e173-4b40-b4a2-3455be004378-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.167416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.167332    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.167416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.167345    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.167416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.167361    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.167416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.167374    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.167416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.167388    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.167416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.167402    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cmfd\" (UniqueName: \"kubernetes.io/projected/22e9f189-e173-4b40-b4a2-3455be004378-kube-api-access-9cmfd\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.167416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.167415    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e9f189-e173-4b40-b4a2-3455be004378-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.167993 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.167429    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.167993 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.167447    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e9f189-e173-4b40-b4a2-3455be004378-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.167993 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.167460    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e9f189-e173-4b40-b4a2-3455be004378-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.168256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.168236    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-envoy" pod "e56d7264-7d35-4dd8-946b-c1ab4d40e941" (UID: "e56d7264-7d35-4dd8-946b-c1ab4d40e941"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:00.168256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.168245    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56d7264-7d35-4dd8-946b-c1ab4d40e941-kube-api-access-98r7z" pod "e56d7264-7d35-4dd8-946b-c1ab4d40e941" (UID: "e56d7264-7d35-4dd8-946b-c1ab4d40e941"). InnerVolumeSpecName "kube-api-access-98r7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:00.168381 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.168364    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-podinfo" pod "e56d7264-7d35-4dd8-946b-c1ab4d40e941" (UID: "e56d7264-7d35-4dd8-946b-c1ab4d40e941"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:51:00.168563 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.168544    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-token" pod "e56d7264-7d35-4dd8-946b-c1ab4d40e941" (UID: "e56d7264-7d35-4dd8-946b-c1ab4d40e941"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:00.269264 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.269230    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.269264 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.269261    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98r7z\" (UniqueName: \"kubernetes.io/projected/e56d7264-7d35-4dd8-946b-c1ab4d40e941-kube-api-access-98r7z\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.269264 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.269271    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.269494 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.269280    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e56d7264-7d35-4dd8-946b-c1ab4d40e941-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:00.782028 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.781645    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:00.796790 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.793199    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main" containerID="cri-o://4339ab245ce4ad4d3a26e1e222f4d9515d112967fc386fa17b37687ed6ea5ea9" gracePeriod=45
Nov 03 19:51:00.813706 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.813644    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main" containerID="cri-o://3142cabe30975b3bba06c15c4c82ef01c97fabe277331d3a9c264c2d5ad67e39" gracePeriod=45
Nov 03 19:51:00.929281 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.929246    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq" event={"ID":"22e9f189-e173-4b40-b4a2-3455be004378","Type":"ContainerDied","Data":"b39852c4d07a5cf2e64eaeb44be5f25ff47eaac24393948b3dff2a29800addf9"}
Nov 03 19:51:00.929412 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.929282    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"
Nov 03 19:51:00.929412 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.929325    5237 scope.go:122] "RemoveContainer" containerID="c932bcc9628f0f0e01ee8e29f48e5d96f4d00e307ac8d0cca6023982b936e188"
Nov 03 19:51:00.933443 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.933420    5237 generic.go:356] "Generic (PLEG): container finished" podID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerID="4339ab245ce4ad4d3a26e1e222f4d9515d112967fc386fa17b37687ed6ea5ea9" exitCode=0
Nov 03 19:51:00.933566 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.933495    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" event={"ID":"c4e11f2e-c05f-4ba6-b748-328c152574a9","Type":"ContainerDied","Data":"4339ab245ce4ad4d3a26e1e222f4d9515d112967fc386fa17b37687ed6ea5ea9"}
Nov 03 19:51:00.935060 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.935040    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"
Nov 03 19:51:00.935173 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.935064    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn" event={"ID":"e56d7264-7d35-4dd8-946b-c1ab4d40e941","Type":"ContainerDied","Data":"3f3da7d2352f771d57a5bcb80cfe67f74a6bbdd8a05353d226e9c2ba3529b8ed"}
Nov 03 19:51:00.950895 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.950872    5237 scope.go:122] "RemoveContainer" containerID="ede69654eeddbd2f71333aacdeaeac2655cd04ffe0046ba709d74a004459b88f"
Nov 03 19:51:00.967086 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.966940    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"]
Nov 03 19:51:00.972908 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.972883    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-7mszq"]
Nov 03 19:51:00.989420 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.989385    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"]
Nov 03 19:51:00.992789 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:00.992764    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-4x9nn"]
Nov 03 19:51:01.131093 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.131064    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:51:01.135382 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.135355    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:51:01.219906 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.219874    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4e11f2e-c05f-4ba6-b748-328c152574a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e11f2e-c05f-4ba6-b748-328c152574a9-tls-certs\") pod \"c4e11f2e-c05f-4ba6-b748-328c152574a9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") "
Nov 03 19:51:01.220405 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.219921    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-model-cache\") pod \"e51fb02e-9d48-4c36-9911-1c077bcc915e\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") "
Nov 03 19:51:01.220405 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.219968    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e11f2e-c05f-4ba6-b748-328c152574a9-kube-api-access-jxzm6\" (UniqueName: \"kubernetes.io/projected/c4e11f2e-c05f-4ba6-b748-328c152574a9-kube-api-access-jxzm6\") pod \"c4e11f2e-c05f-4ba6-b748-328c152574a9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") "
Nov 03 19:51:01.220405 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.220253    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-model-cache" pod "e51fb02e-9d48-4c36-9911-1c077bcc915e" (UID: "e51fb02e-9d48-4c36-9911-1c077bcc915e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:01.220405 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.220255    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-model-cache" pod "c4e11f2e-c05f-4ba6-b748-328c152574a9" (UID: "c4e11f2e-c05f-4ba6-b748-328c152574a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:01.220405 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.220350    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-model-cache\") pod \"c4e11f2e-c05f-4ba6-b748-328c152574a9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") "
Nov 03 19:51:01.220405 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.220408    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-dshm\") pod \"c4e11f2e-c05f-4ba6-b748-328c152574a9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") "
Nov 03 19:51:01.220739 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.220489    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-home\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-home\") pod \"c4e11f2e-c05f-4ba6-b748-328c152574a9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") "
Nov 03 19:51:01.220739 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.220528    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-tmp-dir\") pod \"c4e11f2e-c05f-4ba6-b748-328c152574a9\" (UID: \"c4e11f2e-c05f-4ba6-b748-328c152574a9\") "
Nov 03 19:51:01.220739 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.220557    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e51fb02e-9d48-4c36-9911-1c077bcc915e-tls-certs\" (UniqueName: \"kubernetes.io/secret/e51fb02e-9d48-4c36-9911-1c077bcc915e-tls-certs\") pod \"e51fb02e-9d48-4c36-9911-1c077bcc915e\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") "
Nov 03 19:51:01.220739 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.220624    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-tmp-dir\") pod \"e51fb02e-9d48-4c36-9911-1c077bcc915e\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") "
Nov 03 19:51:01.220739 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.220652    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-home\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-home\") pod \"e51fb02e-9d48-4c36-9911-1c077bcc915e\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") "
Nov 03 19:51:01.220739 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.220706    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-dshm\") pod \"e51fb02e-9d48-4c36-9911-1c077bcc915e\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") "
Nov 03 19:51:01.221072 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.220778    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51fb02e-9d48-4c36-9911-1c077bcc915e-kube-api-access-wnd9r\" (UniqueName: \"kubernetes.io/projected/e51fb02e-9d48-4c36-9911-1c077bcc915e-kube-api-access-wnd9r\") pod \"e51fb02e-9d48-4c36-9911-1c077bcc915e\" (UID: \"e51fb02e-9d48-4c36-9911-1c077bcc915e\") "
Nov 03 19:51:01.221132 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.221078    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-tmp-dir" pod "e51fb02e-9d48-4c36-9911-1c077bcc915e" (UID: "e51fb02e-9d48-4c36-9911-1c077bcc915e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:01.221203 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.221182    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-home" pod "e51fb02e-9d48-4c36-9911-1c077bcc915e" (UID: "e51fb02e-9d48-4c36-9911-1c077bcc915e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:01.221285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.221264    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-home" pod "c4e11f2e-c05f-4ba6-b748-328c152574a9" (UID: "c4e11f2e-c05f-4ba6-b748-328c152574a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:01.221285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.221273    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-tmp-dir" pod "c4e11f2e-c05f-4ba6-b748-328c152574a9" (UID: "c4e11f2e-c05f-4ba6-b748-328c152574a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:01.222372 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.222341    5237 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-home\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:01.222372 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.222374    5237 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-tmp-dir\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:01.222574 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.222389    5237 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-tmp-dir\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:01.222574 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.222402    5237 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-home\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:01.222574 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.222415    5237 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-model-cache\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:01.222574 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.222428    5237 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-model-cache\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:01.224368 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.224279    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e11f2e-c05f-4ba6-b748-328c152574a9-tls-certs" pod "c4e11f2e-c05f-4ba6-b748-328c152574a9" (UID: "c4e11f2e-c05f-4ba6-b748-328c152574a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:51:01.224368 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.224329    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-dshm" pod "e51fb02e-9d48-4c36-9911-1c077bcc915e" (UID: "e51fb02e-9d48-4c36-9911-1c077bcc915e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:01.224557 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.224405    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e11f2e-c05f-4ba6-b748-328c152574a9-kube-api-access-jxzm6" pod "c4e11f2e-c05f-4ba6-b748-328c152574a9" (UID: "c4e11f2e-c05f-4ba6-b748-328c152574a9"). InnerVolumeSpecName "kube-api-access-jxzm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:01.224557 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.224481    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-dshm" pod "c4e11f2e-c05f-4ba6-b748-328c152574a9" (UID: "c4e11f2e-c05f-4ba6-b748-328c152574a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:01.224669 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.224643    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51fb02e-9d48-4c36-9911-1c077bcc915e-kube-api-access-wnd9r" pod "e51fb02e-9d48-4c36-9911-1c077bcc915e" (UID: "e51fb02e-9d48-4c36-9911-1c077bcc915e"). InnerVolumeSpecName "kube-api-access-wnd9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:01.227191 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.227167    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e51fb02e-9d48-4c36-9911-1c077bcc915e-tls-certs" pod "e51fb02e-9d48-4c36-9911-1c077bcc915e" (UID: "e51fb02e-9d48-4c36-9911-1c077bcc915e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:51:01.324300 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.324263    5237 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51fb02e-9d48-4c36-9911-1c077bcc915e-tls-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:01.324300 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.324293    5237 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e51fb02e-9d48-4c36-9911-1c077bcc915e-dshm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:01.324300 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.324306    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnd9r\" (UniqueName: \"kubernetes.io/projected/e51fb02e-9d48-4c36-9911-1c077bcc915e-kube-api-access-wnd9r\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:01.324564 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.324320    5237 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e11f2e-c05f-4ba6-b748-328c152574a9-tls-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:01.324564 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.324330    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxzm6\" (UniqueName: \"kubernetes.io/projected/c4e11f2e-c05f-4ba6-b748-328c152574a9-kube-api-access-jxzm6\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:01.324564 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.324337    5237 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e11f2e-c05f-4ba6-b748-328c152574a9-dshm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:01.679564 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.679468    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:01.861925 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.861894    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0398a799-4e11-4f8e-913f-8a42574c0ae9" path="/var/lib/kubelet/pods/0398a799-4e11-4f8e-913f-8a42574c0ae9/volumes"
Nov 03 19:51:01.862429 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.862416    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22e9f189-e173-4b40-b4a2-3455be004378" path="/var/lib/kubelet/pods/22e9f189-e173-4b40-b4a2-3455be004378/volumes"
Nov 03 19:51:01.862997 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.862982    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e56d7264-7d35-4dd8-946b-c1ab4d40e941" path="/var/lib/kubelet/pods/e56d7264-7d35-4dd8-946b-c1ab4d40e941/volumes"
Nov 03 19:51:01.944712 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.944631    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"
Nov 03 19:51:01.944945 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.944629    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9" event={"ID":"c4e11f2e-c05f-4ba6-b748-328c152574a9","Type":"ContainerDied","Data":"6d2670434f5820fdf0a31602a6e493c3f580bc45ee6fcd3dad24155d3d6e48cb"}
Nov 03 19:51:01.944945 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.944760    5237 scope.go:122] "RemoveContainer" containerID="4339ab245ce4ad4d3a26e1e222f4d9515d112967fc386fa17b37687ed6ea5ea9"
Nov 03 19:51:01.947392 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.947362    5237 generic.go:356] "Generic (PLEG): container finished" podID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerID="3142cabe30975b3bba06c15c4c82ef01c97fabe277331d3a9c264c2d5ad67e39" exitCode=0
Nov 03 19:51:01.947491 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.947391    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" event={"ID":"e51fb02e-9d48-4c36-9911-1c077bcc915e","Type":"ContainerDied","Data":"3142cabe30975b3bba06c15c4c82ef01c97fabe277331d3a9c264c2d5ad67e39"}
Nov 03 19:51:01.947491 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.947425    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw" event={"ID":"e51fb02e-9d48-4c36-9911-1c077bcc915e","Type":"ContainerDied","Data":"22d1b75f4d6716510a9496629f4f790aa054eae6101d27dc7c91c8ecb6f9485b"}
Nov 03 19:51:01.947491 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.947427    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"
Nov 03 19:51:01.955407 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.955382    5237 scope.go:122] "RemoveContainer" containerID="3142cabe30975b3bba06c15c4c82ef01c97fabe277331d3a9c264c2d5ad67e39"
Nov 03 19:51:01.968030 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.967552    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"]
Nov 03 19:51:01.972096 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.972076    5237 scope.go:122] "RemoveContainer" containerID="3142cabe30975b3bba06c15c4c82ef01c97fabe277331d3a9c264c2d5ad67e39"
Nov 03 19:51:01.972578 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:51:01.972553    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3142cabe30975b3bba06c15c4c82ef01c97fabe277331d3a9c264c2d5ad67e39\": container with ID starting with 3142cabe30975b3bba06c15c4c82ef01c97fabe277331d3a9c264c2d5ad67e39 not found: ID does not exist" containerID="3142cabe30975b3bba06c15c4c82ef01c97fabe277331d3a9c264c2d5ad67e39"
Nov 03 19:51:01.972655 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.972592    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3142cabe30975b3bba06c15c4c82ef01c97fabe277331d3a9c264c2d5ad67e39"} err="failed to get container status \"3142cabe30975b3bba06c15c4c82ef01c97fabe277331d3a9c264c2d5ad67e39\": rpc error: code = NotFound desc = could not find container \"3142cabe30975b3bba06c15c4c82ef01c97fabe277331d3a9c264c2d5ad67e39\": container with ID starting with 3142cabe30975b3bba06c15c4c82ef01c97fabe277331d3a9c264c2d5ad67e39 not found: ID does not exist"
Nov 03 19:51:01.979875 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.978032    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ef8f84/simulator-w0-ef8f84-kserve-657ccdf8dd-wxfk9"]
Nov 03 19:51:01.996689 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:01.996654    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"]
Nov 03 19:51:02.000706 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:02.000670    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ef8f84/distinct-2-w0-ef8f84-kserve-569bf84859-rktsw"]
Nov 03 19:51:02.311473 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:02.311432    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:03.041449 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:03.041397    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:03.820989 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:03.820950    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:03.929996 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:03.929959    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" path="/var/lib/kubelet/pods/c4e11f2e-c05f-4ba6-b748-328c152574a9/volumes"
Nov 03 19:51:03.931715 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:03.930784    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" path="/var/lib/kubelet/pods/e51fb02e-9d48-4c36-9911-1c077bcc915e/volumes"
Nov 03 19:51:04.336452 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:04.336413    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:04.481864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:04.481292    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:04.628353 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:04.628258    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:05.092821 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:05.092789    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:05.527878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:05.527760    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:06.275861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:06.254712    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:06.495860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:06.495420    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:06.690624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:06.690579    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:06.923759 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:06.923712    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:07.392413 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:07.392378    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:08.133078 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:08.132962    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:08.268905 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:08.268870    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:08.561190 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:08.561155    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:09.520468 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:09.520435    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:09.652343 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:09.652298    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:10.034131 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:10.034086    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:10.460079 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:10.459986    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:11.236913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:11.236032    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:11.937562 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:11.937503    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:12.724476 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:12.724441    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:12.982101 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:12.982010    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:13.123417 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:13.123380    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:13.264887 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:51:13.264761    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:51:13.297640 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:13.297601    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"]
Nov 03 19:51:13.297925 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:13.297895    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6" podUID="6037ccaa-48d9-450c-8726-213ae6b76500" containerName="istio-proxy" containerID="cri-o://b1acecc213f06a95d54e18185306498c7f752b93a6d4ed10dd5d33b30e2fcabe" gracePeriod=30
Nov 03 19:51:13.368448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:13.368404    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:13.665593 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:13.665555    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:14.692199 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:14.692157    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:15.406534 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:15.406493    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:17.166406 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:17.166370    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:17.374309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:17.374270    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:17.754069 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:17.754031    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:17.896977 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:17.896943    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:18.086640 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.086597    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:18.582678 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.582646    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:51:18.757753 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.757666    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-workload-socket\") pod \"6037ccaa-48d9-450c-8726-213ae6b76500\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") "
Nov 03 19:51:18.757753 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.757726    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-istio-envoy\") pod \"6037ccaa-48d9-450c-8726-213ae6b76500\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") "
Nov 03 19:51:18.758062 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.757823    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-credential-socket\") pod \"6037ccaa-48d9-450c-8726-213ae6b76500\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") "
Nov 03 19:51:18.758062 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.757946    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6037ccaa-48d9-450c-8726-213ae6b76500-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6037ccaa-48d9-450c-8726-213ae6b76500-istiod-ca-cert\") pod \"6037ccaa-48d9-450c-8726-213ae6b76500\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") "
Nov 03 19:51:18.758386 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.758040    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-workload-socket" pod "6037ccaa-48d9-450c-8726-213ae6b76500" (UID: "6037ccaa-48d9-450c-8726-213ae6b76500"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:18.758471 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.758060    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6037ccaa-48d9-450c-8726-213ae6b76500-kube-api-access-lqnj6\" (UniqueName: \"kubernetes.io/projected/6037ccaa-48d9-450c-8726-213ae6b76500-kube-api-access-lqnj6\") pod \"6037ccaa-48d9-450c-8726-213ae6b76500\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") "
Nov 03 19:51:18.758536 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.758131    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-credential-socket" pod "6037ccaa-48d9-450c-8726-213ae6b76500" (UID: "6037ccaa-48d9-450c-8726-213ae6b76500"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:18.758536 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.758353    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6037ccaa-48d9-450c-8726-213ae6b76500-istiod-ca-cert" pod "6037ccaa-48d9-450c-8726-213ae6b76500" (UID: "6037ccaa-48d9-450c-8726-213ae6b76500"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:18.758536 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.758494    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-workload-certs\") pod \"6037ccaa-48d9-450c-8726-213ae6b76500\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") "
Nov 03 19:51:18.758696 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.758558    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6037ccaa-48d9-450c-8726-213ae6b76500-istio-token\" (UniqueName: \"kubernetes.io/projected/6037ccaa-48d9-450c-8726-213ae6b76500-istio-token\") pod \"6037ccaa-48d9-450c-8726-213ae6b76500\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") "
Nov 03 19:51:18.758696 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.758623    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-istio-data\") pod \"6037ccaa-48d9-450c-8726-213ae6b76500\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") "
Nov 03 19:51:18.758696 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.758650    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-workload-certs" pod "6037ccaa-48d9-450c-8726-213ae6b76500" (UID: "6037ccaa-48d9-450c-8726-213ae6b76500"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:18.758696 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.758688    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6037ccaa-48d9-450c-8726-213ae6b76500-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6037ccaa-48d9-450c-8726-213ae6b76500-istio-podinfo\") pod \"6037ccaa-48d9-450c-8726-213ae6b76500\" (UID: \"6037ccaa-48d9-450c-8726-213ae6b76500\") "
Nov 03 19:51:18.758974 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.758824    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-istio-data" pod "6037ccaa-48d9-450c-8726-213ae6b76500" (UID: "6037ccaa-48d9-450c-8726-213ae6b76500"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:18.759492 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.759454    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:18.759492 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.759489    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:18.759679 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.759504    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6037ccaa-48d9-450c-8726-213ae6b76500-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:18.759679 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.759519    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:18.759679 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.759534    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:18.760526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.760491    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-istio-envoy" pod "6037ccaa-48d9-450c-8726-213ae6b76500" (UID: "6037ccaa-48d9-450c-8726-213ae6b76500"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:18.760671 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.760646    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6037ccaa-48d9-450c-8726-213ae6b76500-kube-api-access-lqnj6" pod "6037ccaa-48d9-450c-8726-213ae6b76500" (UID: "6037ccaa-48d9-450c-8726-213ae6b76500"). InnerVolumeSpecName "kube-api-access-lqnj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:18.761087 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.761061    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6037ccaa-48d9-450c-8726-213ae6b76500-istio-podinfo" pod "6037ccaa-48d9-450c-8726-213ae6b76500" (UID: "6037ccaa-48d9-450c-8726-213ae6b76500"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:51:18.761242 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.761218    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6037ccaa-48d9-450c-8726-213ae6b76500-istio-token" pod "6037ccaa-48d9-450c-8726-213ae6b76500" (UID: "6037ccaa-48d9-450c-8726-213ae6b76500"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:18.860805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.860769    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6037ccaa-48d9-450c-8726-213ae6b76500-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:18.860805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.860801    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqnj6\" (UniqueName: \"kubernetes.io/projected/6037ccaa-48d9-450c-8726-213ae6b76500-kube-api-access-lqnj6\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:18.861018 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.860813    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6037ccaa-48d9-450c-8726-213ae6b76500-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:18.861018 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.860824    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6037ccaa-48d9-450c-8726-213ae6b76500-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:18.984114 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:18.984077    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:19.305281 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:19.305245    5237 generic.go:356] "Generic (PLEG): container finished" podID="6037ccaa-48d9-450c-8726-213ae6b76500" containerID="b1acecc213f06a95d54e18185306498c7f752b93a6d4ed10dd5d33b30e2fcabe" exitCode=0
Nov 03 19:51:19.305466 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:19.305305    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"
Nov 03 19:51:19.305466 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:19.305325    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6" event={"ID":"6037ccaa-48d9-450c-8726-213ae6b76500","Type":"ContainerDied","Data":"b1acecc213f06a95d54e18185306498c7f752b93a6d4ed10dd5d33b30e2fcabe"}
Nov 03 19:51:19.305466 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:19.305379    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6" event={"ID":"6037ccaa-48d9-450c-8726-213ae6b76500","Type":"ContainerDied","Data":"5fa2d0e247ea8eda73c78c92440947eb356ad26fe286ee15373f3de3b4703a4d"}
Nov 03 19:51:19.305466 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:19.305405    5237 scope.go:122] "RemoveContainer" containerID="b1acecc213f06a95d54e18185306498c7f752b93a6d4ed10dd5d33b30e2fcabe"
Nov 03 19:51:19.320774 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:19.320751    5237 scope.go:122] "RemoveContainer" containerID="b1acecc213f06a95d54e18185306498c7f752b93a6d4ed10dd5d33b30e2fcabe"
Nov 03 19:51:19.321149 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:51:19.321128    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1acecc213f06a95d54e18185306498c7f752b93a6d4ed10dd5d33b30e2fcabe\": container with ID starting with b1acecc213f06a95d54e18185306498c7f752b93a6d4ed10dd5d33b30e2fcabe not found: ID does not exist" containerID="b1acecc213f06a95d54e18185306498c7f752b93a6d4ed10dd5d33b30e2fcabe"
Nov 03 19:51:19.321223 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:19.321160    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1acecc213f06a95d54e18185306498c7f752b93a6d4ed10dd5d33b30e2fcabe"} err="failed to get container status \"b1acecc213f06a95d54e18185306498c7f752b93a6d4ed10dd5d33b30e2fcabe\": rpc error: code = NotFound desc = could not find container \"b1acecc213f06a95d54e18185306498c7f752b93a6d4ed10dd5d33b30e2fcabe\": container with ID starting with b1acecc213f06a95d54e18185306498c7f752b93a6d4ed10dd5d33b30e2fcabe not found: ID does not exist"
Nov 03 19:51:19.340473 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:19.339503    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"]
Nov 03 19:51:19.347401 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:19.347378    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-hk2t6"]
Nov 03 19:51:19.779499 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:19.779420    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:19.872973 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:19.870191    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6037ccaa-48d9-450c-8726-213ae6b76500" path="/var/lib/kubelet/pods/6037ccaa-48d9-450c-8726-213ae6b76500/volumes"
Nov 03 19:51:20.466357 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:20.466321    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:21.249470 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:21.249431    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:21.654595 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:21.654563    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"]
Nov 03 19:51:21.654811 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:21.654785    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh" podUID="7bde0d49-ee70-4f96-991b-2f38e8984dcf" containerName="maas-api" containerID="cri-o://c890ce84d4dbbc2ce66343e08082f2e5584e14274716901550f3392533e647d9" gracePeriod=30
Nov 03 19:51:21.741055 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:21.741023    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:21.939600 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:21.939512    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:22.113975 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.105916    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:22.146862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.141415    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:51:22.204915 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.204427    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bde0d49-ee70-4f96-991b-2f38e8984dcf-kube-api-access-zmmnn\" (UniqueName: \"kubernetes.io/projected/7bde0d49-ee70-4f96-991b-2f38e8984dcf-kube-api-access-zmmnn\") pod \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\" (UID: \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\") "
Nov 03 19:51:22.204915 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.204514    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bde0d49-ee70-4f96-991b-2f38e8984dcf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bde0d49-ee70-4f96-991b-2f38e8984dcf-metrics-certs\") pod \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\" (UID: \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\") "
Nov 03 19:51:22.204915 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.204660    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bde0d49-ee70-4f96-991b-2f38e8984dcf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bde0d49-ee70-4f96-991b-2f38e8984dcf-openshift-service-ca\") pod \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\" (UID: \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\") "
Nov 03 19:51:22.204915 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.204697    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bde0d49-ee70-4f96-991b-2f38e8984dcf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bde0d49-ee70-4f96-991b-2f38e8984dcf-maas-api-tls\") pod \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\" (UID: \"7bde0d49-ee70-4f96-991b-2f38e8984dcf\") "
Nov 03 19:51:22.211486 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.209470    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bde0d49-ee70-4f96-991b-2f38e8984dcf-openshift-service-ca" pod "7bde0d49-ee70-4f96-991b-2f38e8984dcf" (UID: "7bde0d49-ee70-4f96-991b-2f38e8984dcf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:22.332118 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.316501    5237 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bde0d49-ee70-4f96-991b-2f38e8984dcf-openshift-service-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:22.351886 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.337084    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bde0d49-ee70-4f96-991b-2f38e8984dcf-kube-api-access-zmmnn" pod "7bde0d49-ee70-4f96-991b-2f38e8984dcf" (UID: "7bde0d49-ee70-4f96-991b-2f38e8984dcf"). InnerVolumeSpecName "kube-api-access-zmmnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:22.373398 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.363700    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bde0d49-ee70-4f96-991b-2f38e8984dcf-maas-api-tls" pod "7bde0d49-ee70-4f96-991b-2f38e8984dcf" (UID: "7bde0d49-ee70-4f96-991b-2f38e8984dcf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:51:22.373398 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.364000    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bde0d49-ee70-4f96-991b-2f38e8984dcf-metrics-certs" pod "7bde0d49-ee70-4f96-991b-2f38e8984dcf" (UID: "7bde0d49-ee70-4f96-991b-2f38e8984dcf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:51:22.412206 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.412171    5237 generic.go:356] "Generic (PLEG): container finished" podID="7bde0d49-ee70-4f96-991b-2f38e8984dcf" containerID="c890ce84d4dbbc2ce66343e08082f2e5584e14274716901550f3392533e647d9" exitCode=0
Nov 03 19:51:22.412623 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.412599    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh" event={"ID":"7bde0d49-ee70-4f96-991b-2f38e8984dcf","Type":"ContainerDied","Data":"c890ce84d4dbbc2ce66343e08082f2e5584e14274716901550f3392533e647d9"}
Nov 03 19:51:22.412786 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.412769    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh" event={"ID":"7bde0d49-ee70-4f96-991b-2f38e8984dcf","Type":"ContainerDied","Data":"c2fcfba0e8b61580d0c13f15b05095aa863715f12fe01bbdb9270aa04f4cc6a8"}
Nov 03 19:51:22.412944 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.412928    5237 scope.go:122] "RemoveContainer" containerID="c890ce84d4dbbc2ce66343e08082f2e5584e14274716901550f3392533e647d9"
Nov 03 19:51:22.413180 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.413166    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"
Nov 03 19:51:22.418302 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.418270    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmmnn\" (UniqueName: \"kubernetes.io/projected/7bde0d49-ee70-4f96-991b-2f38e8984dcf-kube-api-access-zmmnn\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:22.418486 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.418469    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bde0d49-ee70-4f96-991b-2f38e8984dcf-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:22.418599 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.418586    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bde0d49-ee70-4f96-991b-2f38e8984dcf-maas-api-tls\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:22.510544 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.510365    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:22.600126 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.600087    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"]
Nov 03 19:51:22.603417 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.603382    5237 scope.go:122] "RemoveContainer" containerID="c890ce84d4dbbc2ce66343e08082f2e5584e14274716901550f3392533e647d9"
Nov 03 19:51:22.604876 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:51:22.604487    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c890ce84d4dbbc2ce66343e08082f2e5584e14274716901550f3392533e647d9\": container with ID starting with c890ce84d4dbbc2ce66343e08082f2e5584e14274716901550f3392533e647d9 not found: ID does not exist" containerID="c890ce84d4dbbc2ce66343e08082f2e5584e14274716901550f3392533e647d9"
Nov 03 19:51:22.604876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.604527    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c890ce84d4dbbc2ce66343e08082f2e5584e14274716901550f3392533e647d9"} err="failed to get container status \"c890ce84d4dbbc2ce66343e08082f2e5584e14274716901550f3392533e647d9\": rpc error: code = NotFound desc = could not find container \"c890ce84d4dbbc2ce66343e08082f2e5584e14274716901550f3392533e647d9\": container with ID starting with c890ce84d4dbbc2ce66343e08082f2e5584e14274716901550f3392533e647d9 not found: ID does not exist"
Nov 03 19:51:22.614198 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.614155    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ef8f84-746f694ccb-kb9sh"]
Nov 03 19:51:22.719028 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:22.718967    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:23.862106 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:23.862074    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bde0d49-ee70-4f96-991b-2f38e8984dcf" path="/var/lib/kubelet/pods/7bde0d49-ee70-4f96-991b-2f38e8984dcf/volumes"
Nov 03 19:51:23.897131 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:23.896961    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:24.178331 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:24.178246    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"]
Nov 03 19:51:24.191279 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:24.191247    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"]
Nov 03 19:51:24.289289 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:24.289251    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:24.445216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:24.445118    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:25.098860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:25.092381    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:25.233017 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:25.232799    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:25.456151 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:25.456047    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:25.747157 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:25.747076    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:26.114054 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:26.114014    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:26.717924 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:26.717889    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:27.337426 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:27.337375    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:27.494729 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:27.494696    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:27.640828 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:27.640746    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:28.058154 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:28.058115    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:28.731319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:28.731285    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:29.178726 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.178678    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb" podUID="a4d0eae8-11ce-483c-97f7-e0aef82ed54b" containerName="payload-processing" containerID="cri-o://1c9d739eb58e78a7ec8d8f9431ba30a1beea932f7ffb1068a924029b1c58c6ff" gracePeriod=25
Nov 03 19:51:29.192247 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.192210    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv" podUID="e0e9e459-f64b-44b6-a217-56d4e11feada" containerName="payload-pre-processing" containerID="cri-o://473de8872a719b7f9acef52866d9867c2e0e6340656ce348f1674a4db1dd1563" gracePeriod=25
Nov 03 19:51:29.502450 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.502413    5237 generic.go:356] "Generic (PLEG): container finished" podID="a4d0eae8-11ce-483c-97f7-e0aef82ed54b" containerID="1c9d739eb58e78a7ec8d8f9431ba30a1beea932f7ffb1068a924029b1c58c6ff" exitCode=0
Nov 03 19:51:29.502611 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.502584    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb" event={"ID":"a4d0eae8-11ce-483c-97f7-e0aef82ed54b","Type":"ContainerDied","Data":"1c9d739eb58e78a7ec8d8f9431ba30a1beea932f7ffb1068a924029b1c58c6ff"}
Nov 03 19:51:29.502685 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.502637    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb" event={"ID":"a4d0eae8-11ce-483c-97f7-e0aef82ed54b","Type":"ContainerDied","Data":"e50c9aac46baad21ba8873c28d2c9f22a70643b391e2f2cc7e4576f15f79405e"}
Nov 03 19:51:29.502685 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.502667    5237 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e50c9aac46baad21ba8873c28d2c9f22a70643b391e2f2cc7e4576f15f79405e"
Nov 03 19:51:29.504921 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.504897    5237 generic.go:356] "Generic (PLEG): container finished" podID="e0e9e459-f64b-44b6-a217-56d4e11feada" containerID="473de8872a719b7f9acef52866d9867c2e0e6340656ce348f1674a4db1dd1563" exitCode=0
Nov 03 19:51:29.505049 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.504938    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv" event={"ID":"e0e9e459-f64b-44b6-a217-56d4e11feada","Type":"ContainerDied","Data":"473de8872a719b7f9acef52866d9867c2e0e6340656ce348f1674a4db1dd1563"}
Nov 03 19:51:29.507128 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.507109    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"
Nov 03 19:51:29.532558 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.532533    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"
Nov 03 19:51:29.561881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.561810    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e9e459-f64b-44b6-a217-56d4e11feada-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e9e459-f64b-44b6-a217-56d4e11feada-plugins-config-volume\") pod \"e0e9e459-f64b-44b6-a217-56d4e11feada\" (UID: \"e0e9e459-f64b-44b6-a217-56d4e11feada\") "
Nov 03 19:51:29.562068 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.561981    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d0eae8-11ce-483c-97f7-e0aef82ed54b-kube-api-access-gwqx9\" (UniqueName: \"kubernetes.io/projected/a4d0eae8-11ce-483c-97f7-e0aef82ed54b-kube-api-access-gwqx9\") pod \"a4d0eae8-11ce-483c-97f7-e0aef82ed54b\" (UID: \"a4d0eae8-11ce-483c-97f7-e0aef82ed54b\") "
Nov 03 19:51:29.562186 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.562166    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e9e459-f64b-44b6-a217-56d4e11feada-kube-api-access-27rjx\" (UniqueName: \"kubernetes.io/projected/e0e9e459-f64b-44b6-a217-56d4e11feada-kube-api-access-27rjx\") pod \"e0e9e459-f64b-44b6-a217-56d4e11feada\" (UID: \"e0e9e459-f64b-44b6-a217-56d4e11feada\") "
Nov 03 19:51:29.562247 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.562210    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4d0eae8-11ce-483c-97f7-e0aef82ed54b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4d0eae8-11ce-483c-97f7-e0aef82ed54b-plugins-config-volume\") pod \"a4d0eae8-11ce-483c-97f7-e0aef82ed54b\" (UID: \"a4d0eae8-11ce-483c-97f7-e0aef82ed54b\") "
Nov 03 19:51:29.562650 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.562628    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e9e459-f64b-44b6-a217-56d4e11feada-plugins-config-volume" pod "e0e9e459-f64b-44b6-a217-56d4e11feada" (UID: "e0e9e459-f64b-44b6-a217-56d4e11feada"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:29.563090 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.563069    5237 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e9e459-f64b-44b6-a217-56d4e11feada-plugins-config-volume\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:29.563224 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.563102    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d0eae8-11ce-483c-97f7-e0aef82ed54b-plugins-config-volume" pod "a4d0eae8-11ce-483c-97f7-e0aef82ed54b" (UID: "a4d0eae8-11ce-483c-97f7-e0aef82ed54b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:29.565514 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.565491    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e9e459-f64b-44b6-a217-56d4e11feada-kube-api-access-27rjx" pod "e0e9e459-f64b-44b6-a217-56d4e11feada" (UID: "e0e9e459-f64b-44b6-a217-56d4e11feada"). InnerVolumeSpecName "kube-api-access-27rjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:29.567779 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.567757    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d0eae8-11ce-483c-97f7-e0aef82ed54b-kube-api-access-gwqx9" pod "a4d0eae8-11ce-483c-97f7-e0aef82ed54b" (UID: "a4d0eae8-11ce-483c-97f7-e0aef82ed54b"). InnerVolumeSpecName "kube-api-access-gwqx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:29.642761 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.642725    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:29.664662 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.664608    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27rjx\" (UniqueName: \"kubernetes.io/projected/e0e9e459-f64b-44b6-a217-56d4e11feada-kube-api-access-27rjx\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:29.664662 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.664644    5237 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4d0eae8-11ce-483c-97f7-e0aef82ed54b-plugins-config-volume\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:29.664662 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:29.664659    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwqx9\" (UniqueName: \"kubernetes.io/projected/a4d0eae8-11ce-483c-97f7-e0aef82ed54b-kube-api-access-gwqx9\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:30.236338 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.236297    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:30.393756 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.393714    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:30.514151 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.514063    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"
Nov 03 19:51:30.514151 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.514084    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv" event={"ID":"e0e9e459-f64b-44b6-a217-56d4e11feada","Type":"ContainerDied","Data":"9d031f34137d1af69f826294ba5816eef5531bbc26118e8981827bc01db7d29e"}
Nov 03 19:51:30.514399 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.514185    5237 scope.go:122] "RemoveContainer" containerID="473de8872a719b7f9acef52866d9867c2e0e6340656ce348f1674a4db1dd1563"
Nov 03 19:51:30.514399 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.514266    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"
Nov 03 19:51:30.559319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.559277    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"]
Nov 03 19:51:30.564019 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.563580    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:30.566929 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.566387    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5444b0-78fd4487cf-j29jv"]
Nov 03 19:51:30.600485 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.585660    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"]
Nov 03 19:51:30.600485 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.592773    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"]
Nov 03 19:51:30.600485 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.593050    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" containerName="istio-proxy" containerID="cri-o://1bee9505297b475498c6691a54a9fe4fb618fe5b4438320854bee1b2728f6226" gracePeriod=30
Nov 03 19:51:30.600485 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.600336    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5444b0-df5d58595-7t6rb"]
Nov 03 19:51:30.603494 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.603467    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"]
Nov 03 19:51:30.603761 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.603729    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy" containerID="cri-o://f72f1dad6fc491872ef0294081f6f0c7bd7b8a35abcdf98cdb4bc3da8fad8ede" gracePeriod=30
Nov 03 19:51:30.608705 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.608679    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"]
Nov 03 19:51:30.609004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.608946    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q" podUID="941d6f5f-47cd-439e-9463-ecc633799b05" containerName="istio-proxy" containerID="cri-o://799963e23028ff1be61f73a5f7e79717bc8f8a6f9eb83ff9c19d2ae59aa5cdd4" gracePeriod=30
Nov 03 19:51:30.766919 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:30.727989    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:31.616591 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:31.616553    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:31.862899 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:31.862866    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4d0eae8-11ce-483c-97f7-e0aef82ed54b" path="/var/lib/kubelet/pods/a4d0eae8-11ce-483c-97f7-e0aef82ed54b/volumes"
Nov 03 19:51:31.863487 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:31.863461    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e9e459-f64b-44b6-a217-56d4e11feada" path="/var/lib/kubelet/pods/e0e9e459-f64b-44b6-a217-56d4e11feada/volumes"
Nov 03 19:51:32.862931 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:32.862893    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:33.497647 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:33.497583    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:33.748035 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:33.747879    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:33.770043 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:33.770008    5237 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:51:33.771392 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:33.771356    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q" podUID="941d6f5f-47cd-439e-9463-ecc633799b05" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:51:34.086873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:34.083682    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:34.287861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:34.275044    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:34.602718 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:34.602680    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:34.747248 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:34.747163    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:35.265664 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:35.265630    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:35.501853 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:35.501798    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:35.934342 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:35.934316    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:51:35.956958 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:35.956933    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:51:36.007217 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007187    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/941d6f5f-47cd-439e-9463-ecc633799b05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941d6f5f-47cd-439e-9463-ecc633799b05-istiod-ca-cert\") pod \"941d6f5f-47cd-439e-9463-ecc633799b05\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") "
Nov 03 19:51:36.007417 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007231    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-workload-socket\") pod \"941d6f5f-47cd-439e-9463-ecc633799b05\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") "
Nov 03 19:51:36.007417 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007291    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-istio-envoy\") pod \"27821d19-33d6-4db9-a259-a7a555d03e15\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") "
Nov 03 19:51:36.007417 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007322    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-istio-envoy\") pod \"941d6f5f-47cd-439e-9463-ecc633799b05\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") "
Nov 03 19:51:36.007417 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007349    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-workload-certs\") pod \"941d6f5f-47cd-439e-9463-ecc633799b05\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") "
Nov 03 19:51:36.007417 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007384    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-istio-data\") pod \"941d6f5f-47cd-439e-9463-ecc633799b05\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") "
Nov 03 19:51:36.007417 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007417    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27821d19-33d6-4db9-a259-a7a555d03e15-istio-token\" (UniqueName: \"kubernetes.io/projected/27821d19-33d6-4db9-a259-a7a555d03e15-istio-token\") pod \"27821d19-33d6-4db9-a259-a7a555d03e15\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") "
Nov 03 19:51:36.007748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007469    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/941d6f5f-47cd-439e-9463-ecc633799b05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941d6f5f-47cd-439e-9463-ecc633799b05-istio-podinfo\") pod \"941d6f5f-47cd-439e-9463-ecc633799b05\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") "
Nov 03 19:51:36.007748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007532    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-workload-socket\") pod \"27821d19-33d6-4db9-a259-a7a555d03e15\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") "
Nov 03 19:51:36.007748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007624    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941d6f5f-47cd-439e-9463-ecc633799b05-kube-api-access-rph4z\" (UniqueName: \"kubernetes.io/projected/941d6f5f-47cd-439e-9463-ecc633799b05-kube-api-access-rph4z\") pod \"941d6f5f-47cd-439e-9463-ecc633799b05\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") "
Nov 03 19:51:36.007748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007679    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-credential-socket\") pod \"27821d19-33d6-4db9-a259-a7a555d03e15\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") "
Nov 03 19:51:36.008017 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007757    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27821d19-33d6-4db9-a259-a7a555d03e15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27821d19-33d6-4db9-a259-a7a555d03e15-istiod-ca-cert\") pod \"27821d19-33d6-4db9-a259-a7a555d03e15\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") "
Nov 03 19:51:36.008017 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007759    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-workload-certs" pod "941d6f5f-47cd-439e-9463-ecc633799b05" (UID: "941d6f5f-47cd-439e-9463-ecc633799b05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:36.008017 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007853    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-workload-certs\") pod \"27821d19-33d6-4db9-a259-a7a555d03e15\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") "
Nov 03 19:51:36.008017 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007866    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941d6f5f-47cd-439e-9463-ecc633799b05-istiod-ca-cert" pod "941d6f5f-47cd-439e-9463-ecc633799b05" (UID: "941d6f5f-47cd-439e-9463-ecc633799b05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:36.008017 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007960    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-workload-socket" pod "27821d19-33d6-4db9-a259-a7a555d03e15" (UID: "27821d19-33d6-4db9-a259-a7a555d03e15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:36.008017 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007971    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27821d19-33d6-4db9-a259-a7a555d03e15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27821d19-33d6-4db9-a259-a7a555d03e15-istio-podinfo\") pod \"27821d19-33d6-4db9-a259-a7a555d03e15\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") "
Nov 03 19:51:36.008391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.008020    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27821d19-33d6-4db9-a259-a7a555d03e15-kube-api-access-45chc\" (UniqueName: \"kubernetes.io/projected/27821d19-33d6-4db9-a259-a7a555d03e15-kube-api-access-45chc\") pod \"27821d19-33d6-4db9-a259-a7a555d03e15\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") "
Nov 03 19:51:36.008391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.008058    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-credential-socket\") pod \"941d6f5f-47cd-439e-9463-ecc633799b05\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") "
Nov 03 19:51:36.008391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.007769    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-istio-data" pod "941d6f5f-47cd-439e-9463-ecc633799b05" (UID: "941d6f5f-47cd-439e-9463-ecc633799b05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:36.008391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.008100    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27821d19-33d6-4db9-a259-a7a555d03e15-istiod-ca-cert" pod "27821d19-33d6-4db9-a259-a7a555d03e15" (UID: "27821d19-33d6-4db9-a259-a7a555d03e15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:36.008391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.008182    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-istio-data\") pod \"27821d19-33d6-4db9-a259-a7a555d03e15\" (UID: \"27821d19-33d6-4db9-a259-a7a555d03e15\") "
Nov 03 19:51:36.008391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.008219    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941d6f5f-47cd-439e-9463-ecc633799b05-istio-token\" (UniqueName: \"kubernetes.io/projected/941d6f5f-47cd-439e-9463-ecc633799b05-istio-token\") pod \"941d6f5f-47cd-439e-9463-ecc633799b05\" (UID: \"941d6f5f-47cd-439e-9463-ecc633799b05\") "
Nov 03 19:51:36.008391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.008288    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-workload-certs" pod "27821d19-33d6-4db9-a259-a7a555d03e15" (UID: "27821d19-33d6-4db9-a259-a7a555d03e15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:36.008391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.008317    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-workload-socket" pod "941d6f5f-47cd-439e-9463-ecc633799b05" (UID: "941d6f5f-47cd-439e-9463-ecc633799b05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:36.008889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.008493    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-credential-socket" pod "27821d19-33d6-4db9-a259-a7a555d03e15" (UID: "27821d19-33d6-4db9-a259-a7a555d03e15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:36.009092 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.009064    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.009256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.009226    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27821d19-33d6-4db9-a259-a7a555d03e15-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.009371 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.009357    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.009866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.009598    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-istio-data" pod "27821d19-33d6-4db9-a259-a7a555d03e15" (UID: "27821d19-33d6-4db9-a259-a7a555d03e15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:36.009866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.009729    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-credential-socket" pod "941d6f5f-47cd-439e-9463-ecc633799b05" (UID: "941d6f5f-47cd-439e-9463-ecc633799b05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:36.010088 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.010070    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941d6f5f-47cd-439e-9463-ecc633799b05-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.010188 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.010173    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.010281 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.010269    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.010355 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.010340    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.010448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.010358    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.011319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.011291    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-istio-envoy" pod "27821d19-33d6-4db9-a259-a7a555d03e15" (UID: "27821d19-33d6-4db9-a259-a7a555d03e15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:36.011458 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.011433    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-istio-envoy" pod "941d6f5f-47cd-439e-9463-ecc633799b05" (UID: "941d6f5f-47cd-439e-9463-ecc633799b05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:36.011584 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.011559    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/941d6f5f-47cd-439e-9463-ecc633799b05-istio-podinfo" pod "941d6f5f-47cd-439e-9463-ecc633799b05" (UID: "941d6f5f-47cd-439e-9463-ecc633799b05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:51:36.011666 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.011612    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941d6f5f-47cd-439e-9463-ecc633799b05-kube-api-access-rph4z" pod "941d6f5f-47cd-439e-9463-ecc633799b05" (UID: "941d6f5f-47cd-439e-9463-ecc633799b05"). InnerVolumeSpecName "kube-api-access-rph4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:36.011739 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.011683    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27821d19-33d6-4db9-a259-a7a555d03e15-istio-token" pod "27821d19-33d6-4db9-a259-a7a555d03e15" (UID: "27821d19-33d6-4db9-a259-a7a555d03e15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:36.011876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.011792    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27821d19-33d6-4db9-a259-a7a555d03e15-istio-podinfo" pod "27821d19-33d6-4db9-a259-a7a555d03e15" (UID: "27821d19-33d6-4db9-a259-a7a555d03e15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:51:36.012201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.012175    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27821d19-33d6-4db9-a259-a7a555d03e15-kube-api-access-45chc" pod "27821d19-33d6-4db9-a259-a7a555d03e15" (UID: "27821d19-33d6-4db9-a259-a7a555d03e15"). InnerVolumeSpecName "kube-api-access-45chc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:36.012710 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.012693    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941d6f5f-47cd-439e-9463-ecc633799b05-istio-token" pod "941d6f5f-47cd-439e-9463-ecc633799b05" (UID: "941d6f5f-47cd-439e-9463-ecc633799b05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:36.111408 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.111370    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27821d19-33d6-4db9-a259-a7a555d03e15-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.111408 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.111403    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941d6f5f-47cd-439e-9463-ecc633799b05-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.111408 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.111414    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rph4z\" (UniqueName: \"kubernetes.io/projected/941d6f5f-47cd-439e-9463-ecc633799b05-kube-api-access-rph4z\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.111652 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.111423    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27821d19-33d6-4db9-a259-a7a555d03e15-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.111652 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.111433    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45chc\" (UniqueName: \"kubernetes.io/projected/27821d19-33d6-4db9-a259-a7a555d03e15-kube-api-access-45chc\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.111652 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.111442    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.111652 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.111452    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.111652 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.111460    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/941d6f5f-47cd-439e-9463-ecc633799b05-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.111652 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.111467    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27821d19-33d6-4db9-a259-a7a555d03e15-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.111652 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.111474    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941d6f5f-47cd-439e-9463-ecc633799b05-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.270567 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.270456    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:36.344745 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.344721    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.424378    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-envoy\") pod \"9c1faf58-6636-4c3c-a702-53c2fe91d649\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") "
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.424453    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-credential-socket\") pod \"9c1faf58-6636-4c3c-a702-53c2fe91d649\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") "
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.424809    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-credential-socket" pod "9c1faf58-6636-4c3c-a702-53c2fe91d649" (UID: "9c1faf58-6636-4c3c-a702-53c2fe91d649"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.424920    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-workload-certs\") pod \"9c1faf58-6636-4c3c-a702-53c2fe91d649\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") "
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.427455    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-workload-certs" pod "9c1faf58-6636-4c3c-a702-53c2fe91d649" (UID: "9c1faf58-6636-4c3c-a702-53c2fe91d649"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.427596    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1faf58-6636-4c3c-a702-53c2fe91d649-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1faf58-6636-4c3c-a702-53c2fe91d649-istiod-ca-cert\") pod \"9c1faf58-6636-4c3c-a702-53c2fe91d649\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") "
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.427716    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1faf58-6636-4c3c-a702-53c2fe91d649-kube-api-access-wlrg7\" (UniqueName: \"kubernetes.io/projected/9c1faf58-6636-4c3c-a702-53c2fe91d649-kube-api-access-wlrg7\") pod \"9c1faf58-6636-4c3c-a702-53c2fe91d649\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") "
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.428073    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1faf58-6636-4c3c-a702-53c2fe91d649-istiod-ca-cert" pod "9c1faf58-6636-4c3c-a702-53c2fe91d649" (UID: "9c1faf58-6636-4c3c-a702-53c2fe91d649"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.428124    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-workload-socket\") pod \"9c1faf58-6636-4c3c-a702-53c2fe91d649\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") "
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.428282    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-podinfo\") pod \"9c1faf58-6636-4c3c-a702-53c2fe91d649\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") "
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.428404    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-data\") pod \"9c1faf58-6636-4c3c-a702-53c2fe91d649\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") "
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.428433    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-token\" (UniqueName: \"kubernetes.io/projected/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-token\") pod \"9c1faf58-6636-4c3c-a702-53c2fe91d649\" (UID: \"9c1faf58-6636-4c3c-a702-53c2fe91d649\") "
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.428616    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-workload-socket" pod "9c1faf58-6636-4c3c-a702-53c2fe91d649" (UID: "9c1faf58-6636-4c3c-a702-53c2fe91d649"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.429295    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.429318    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.429332    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.429350    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1faf58-6636-4c3c-a702-53c2fe91d649-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.431724    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:36.434415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.432472    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-envoy" pod "9c1faf58-6636-4c3c-a702-53c2fe91d649" (UID: "9c1faf58-6636-4c3c-a702-53c2fe91d649"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:36.438657 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.438615    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-token" pod "9c1faf58-6636-4c3c-a702-53c2fe91d649" (UID: "9c1faf58-6636-4c3c-a702-53c2fe91d649"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:36.449035 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.441829    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-data" pod "9c1faf58-6636-4c3c-a702-53c2fe91d649" (UID: "9c1faf58-6636-4c3c-a702-53c2fe91d649"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:36.449035 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.448991    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1faf58-6636-4c3c-a702-53c2fe91d649-kube-api-access-wlrg7" pod "9c1faf58-6636-4c3c-a702-53c2fe91d649" (UID: "9c1faf58-6636-4c3c-a702-53c2fe91d649"). InnerVolumeSpecName "kube-api-access-wlrg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:36.459393 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.458387    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-podinfo" pod "9c1faf58-6636-4c3c-a702-53c2fe91d649" (UID: "9c1faf58-6636-4c3c-a702-53c2fe91d649"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:51:36.531359 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.531317    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.531359 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.531354    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.531359 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.531368    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.531623 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.531381    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlrg7\" (UniqueName: \"kubernetes.io/projected/9c1faf58-6636-4c3c-a702-53c2fe91d649-kube-api-access-wlrg7\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.531623 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.531395    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1faf58-6636-4c3c-a702-53c2fe91d649-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:36.579293 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.579254    5237 generic.go:356] "Generic (PLEG): container finished" podID="941d6f5f-47cd-439e-9463-ecc633799b05" containerID="799963e23028ff1be61f73a5f7e79717bc8f8a6f9eb83ff9c19d2ae59aa5cdd4" exitCode=0
Nov 03 19:51:36.579489 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.579314    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"
Nov 03 19:51:36.579489 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.579331    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q" event={"ID":"941d6f5f-47cd-439e-9463-ecc633799b05","Type":"ContainerDied","Data":"799963e23028ff1be61f73a5f7e79717bc8f8a6f9eb83ff9c19d2ae59aa5cdd4"}
Nov 03 19:51:36.579489 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.579385    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q" event={"ID":"941d6f5f-47cd-439e-9463-ecc633799b05","Type":"ContainerDied","Data":"4d18194edd45fa78249e4a4f48ffd7f43b23e24a8ce604b4e1b4ae6ec07816fb"}
Nov 03 19:51:36.579489 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.579409    5237 scope.go:122] "RemoveContainer" containerID="799963e23028ff1be61f73a5f7e79717bc8f8a6f9eb83ff9c19d2ae59aa5cdd4"
Nov 03 19:51:36.581083 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.581049    5237 generic.go:356] "Generic (PLEG): container finished" podID="27821d19-33d6-4db9-a259-a7a555d03e15" containerID="1bee9505297b475498c6691a54a9fe4fb618fe5b4438320854bee1b2728f6226" exitCode=0
Nov 03 19:51:36.581221 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.581124    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"
Nov 03 19:51:36.581221 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.581179    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" event={"ID":"27821d19-33d6-4db9-a259-a7a555d03e15","Type":"ContainerDied","Data":"1bee9505297b475498c6691a54a9fe4fb618fe5b4438320854bee1b2728f6226"}
Nov 03 19:51:36.581342 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.581234    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f" event={"ID":"27821d19-33d6-4db9-a259-a7a555d03e15","Type":"ContainerDied","Data":"452bd1692a8299225875010d0e153c00e8302a4afaf175c3418257ce7e5f8e52"}
Nov 03 19:51:36.582863 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.582811    5237 generic.go:356] "Generic (PLEG): container finished" podID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerID="f72f1dad6fc491872ef0294081f6f0c7bd7b8a35abcdf98cdb4bc3da8fad8ede" exitCode=0
Nov 03 19:51:36.582962 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.582892    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"
Nov 03 19:51:36.582962 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.582942    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" event={"ID":"9c1faf58-6636-4c3c-a702-53c2fe91d649","Type":"ContainerDied","Data":"f72f1dad6fc491872ef0294081f6f0c7bd7b8a35abcdf98cdb4bc3da8fad8ede"}
Nov 03 19:51:36.583053 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.582990    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss" event={"ID":"9c1faf58-6636-4c3c-a702-53c2fe91d649","Type":"ContainerDied","Data":"3d94939e6b623df883af7ecb5f7089719da62b20e713c035214bb3407069b187"}
Nov 03 19:51:36.592437 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.592413    5237 scope.go:122] "RemoveContainer" containerID="799963e23028ff1be61f73a5f7e79717bc8f8a6f9eb83ff9c19d2ae59aa5cdd4"
Nov 03 19:51:36.592728 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:51:36.592709    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799963e23028ff1be61f73a5f7e79717bc8f8a6f9eb83ff9c19d2ae59aa5cdd4\": container with ID starting with 799963e23028ff1be61f73a5f7e79717bc8f8a6f9eb83ff9c19d2ae59aa5cdd4 not found: ID does not exist" containerID="799963e23028ff1be61f73a5f7e79717bc8f8a6f9eb83ff9c19d2ae59aa5cdd4"
Nov 03 19:51:36.592783 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.592737    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799963e23028ff1be61f73a5f7e79717bc8f8a6f9eb83ff9c19d2ae59aa5cdd4"} err="failed to get container status \"799963e23028ff1be61f73a5f7e79717bc8f8a6f9eb83ff9c19d2ae59aa5cdd4\": rpc error: code = NotFound desc = could not find container \"799963e23028ff1be61f73a5f7e79717bc8f8a6f9eb83ff9c19d2ae59aa5cdd4\": container with ID starting with 799963e23028ff1be61f73a5f7e79717bc8f8a6f9eb83ff9c19d2ae59aa5cdd4 not found: ID does not exist"
Nov 03 19:51:36.592783 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.592756    5237 scope.go:122] "RemoveContainer" containerID="1bee9505297b475498c6691a54a9fe4fb618fe5b4438320854bee1b2728f6226"
Nov 03 19:51:36.602493 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.602467    5237 scope.go:122] "RemoveContainer" containerID="1bee9505297b475498c6691a54a9fe4fb618fe5b4438320854bee1b2728f6226"
Nov 03 19:51:36.602796 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:51:36.602769    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bee9505297b475498c6691a54a9fe4fb618fe5b4438320854bee1b2728f6226\": container with ID starting with 1bee9505297b475498c6691a54a9fe4fb618fe5b4438320854bee1b2728f6226 not found: ID does not exist" containerID="1bee9505297b475498c6691a54a9fe4fb618fe5b4438320854bee1b2728f6226"
Nov 03 19:51:36.602870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.602808    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bee9505297b475498c6691a54a9fe4fb618fe5b4438320854bee1b2728f6226"} err="failed to get container status \"1bee9505297b475498c6691a54a9fe4fb618fe5b4438320854bee1b2728f6226\": rpc error: code = NotFound desc = could not find container \"1bee9505297b475498c6691a54a9fe4fb618fe5b4438320854bee1b2728f6226\": container with ID starting with 1bee9505297b475498c6691a54a9fe4fb618fe5b4438320854bee1b2728f6226 not found: ID does not exist"
Nov 03 19:51:36.602870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.602830    5237 scope.go:122] "RemoveContainer" containerID="f72f1dad6fc491872ef0294081f6f0c7bd7b8a35abcdf98cdb4bc3da8fad8ede"
Nov 03 19:51:36.613069 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.613050    5237 scope.go:122] "RemoveContainer" containerID="f72f1dad6fc491872ef0294081f6f0c7bd7b8a35abcdf98cdb4bc3da8fad8ede"
Nov 03 19:51:36.613356 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:51:36.613330    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f72f1dad6fc491872ef0294081f6f0c7bd7b8a35abcdf98cdb4bc3da8fad8ede\": container with ID starting with f72f1dad6fc491872ef0294081f6f0c7bd7b8a35abcdf98cdb4bc3da8fad8ede not found: ID does not exist" containerID="f72f1dad6fc491872ef0294081f6f0c7bd7b8a35abcdf98cdb4bc3da8fad8ede"
Nov 03 19:51:36.613445 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.613359    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f72f1dad6fc491872ef0294081f6f0c7bd7b8a35abcdf98cdb4bc3da8fad8ede"} err="failed to get container status \"f72f1dad6fc491872ef0294081f6f0c7bd7b8a35abcdf98cdb4bc3da8fad8ede\": rpc error: code = NotFound desc = could not find container \"f72f1dad6fc491872ef0294081f6f0c7bd7b8a35abcdf98cdb4bc3da8fad8ede\": container with ID starting with f72f1dad6fc491872ef0294081f6f0c7bd7b8a35abcdf98cdb4bc3da8fad8ede not found: ID does not exist"
Nov 03 19:51:36.620310 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.620279    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"]
Nov 03 19:51:36.636648 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.636618    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-drh5q"]
Nov 03 19:51:36.689728 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.689641    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"]
Nov 03 19:51:36.689728 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.689703    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-nbf6f"]
Nov 03 19:51:36.700017 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.699984    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"]
Nov 03 19:51:36.704232 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.704203    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-5wlss"]
Nov 03 19:51:36.755040 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:36.755009    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:37.071773 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:37.071742    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:37.286044 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:37.285999    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:37.919225 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:37.919178    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" path="/var/lib/kubelet/pods/27821d19-33d6-4db9-a259-a7a555d03e15/volumes"
Nov 03 19:51:37.920675 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:37.920619    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="941d6f5f-47cd-439e-9463-ecc633799b05" path="/var/lib/kubelet/pods/941d6f5f-47cd-439e-9463-ecc633799b05/volumes"
Nov 03 19:51:37.921581 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:37.921556    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" path="/var/lib/kubelet/pods/9c1faf58-6636-4c3c-a702-53c2fe91d649/volumes"
Nov 03 19:51:38.110075 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:38.110021    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:38.240670 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:38.240548    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:38.712754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:38.712717    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:39.853308 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:39.853262    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:40.115913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:40.115497    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:40.736760 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:40.736724    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"]
Nov 03 19:51:40.959197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:40.959164    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:41.106099 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:41.106064    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:42.172180 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:42.172144    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:42.256014 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:42.255984    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"]
Nov 03 19:51:42.282861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:42.282810    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"]
Nov 03 19:51:42.420555 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:42.420520    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:43.075860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:43.075675    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:43.219917 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:43.216466    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:43.372422 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:51:43.372343    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:51:43.376037 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:43.376007    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:43.549553 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:43.549515    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:43.697985 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:43.690595    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:43.864884 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:43.864380    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:44.714131 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:44.714080    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:45.527569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:45.527531    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:45.737625 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:45.737553    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr" podUID="af9b6d3c-e4c9-478c-9d19-36444a7c4e77" containerName="payload-processing" containerID="cri-o://aef580658217fa9e7afb2f3b3555c062a4d1e1efa85b68e99b8abde0bb95d4c1" gracePeriod=25
Nov 03 19:51:46.021355 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:46.021323    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"
Nov 03 19:51:46.073795 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:46.073764    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9b6d3c-e4c9-478c-9d19-36444a7c4e77-kube-api-access-j7x7p\" (UniqueName: \"kubernetes.io/projected/af9b6d3c-e4c9-478c-9d19-36444a7c4e77-kube-api-access-j7x7p\") pod \"af9b6d3c-e4c9-478c-9d19-36444a7c4e77\" (UID: \"af9b6d3c-e4c9-478c-9d19-36444a7c4e77\") "
Nov 03 19:51:46.074061 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:46.074047    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af9b6d3c-e4c9-478c-9d19-36444a7c4e77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9b6d3c-e4c9-478c-9d19-36444a7c4e77-plugins-config-volume\") pod \"af9b6d3c-e4c9-478c-9d19-36444a7c4e77\" (UID: \"af9b6d3c-e4c9-478c-9d19-36444a7c4e77\") "
Nov 03 19:51:46.074435 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:46.074410    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9b6d3c-e4c9-478c-9d19-36444a7c4e77-plugins-config-volume" pod "af9b6d3c-e4c9-478c-9d19-36444a7c4e77" (UID: "af9b6d3c-e4c9-478c-9d19-36444a7c4e77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:46.076168 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:46.076134    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9b6d3c-e4c9-478c-9d19-36444a7c4e77-kube-api-access-j7x7p" pod "af9b6d3c-e4c9-478c-9d19-36444a7c4e77" (UID: "af9b6d3c-e4c9-478c-9d19-36444a7c4e77"). InnerVolumeSpecName "kube-api-access-j7x7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:46.176201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:46.176118    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7x7p\" (UniqueName: \"kubernetes.io/projected/af9b6d3c-e4c9-478c-9d19-36444a7c4e77-kube-api-access-j7x7p\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:46.176201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:46.176150    5237 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9b6d3c-e4c9-478c-9d19-36444a7c4e77-plugins-config-volume\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:46.218325 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:46.218284    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:46.726130 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:46.726097    5237 generic.go:356] "Generic (PLEG): container finished" podID="af9b6d3c-e4c9-478c-9d19-36444a7c4e77" containerID="aef580658217fa9e7afb2f3b3555c062a4d1e1efa85b68e99b8abde0bb95d4c1" exitCode=0
Nov 03 19:51:46.726305 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:46.726144    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"
Nov 03 19:51:46.726305 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:46.726172    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr" event={"ID":"af9b6d3c-e4c9-478c-9d19-36444a7c4e77","Type":"ContainerDied","Data":"aef580658217fa9e7afb2f3b3555c062a4d1e1efa85b68e99b8abde0bb95d4c1"}
Nov 03 19:51:46.726305 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:46.726226    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr" event={"ID":"af9b6d3c-e4c9-478c-9d19-36444a7c4e77","Type":"ContainerDied","Data":"1151e55043915b8adaee86195ce323ece9713b05ef687d807ae65346ad5052df"}
Nov 03 19:51:46.726305 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:46.726248    5237 scope.go:122] "RemoveContainer" containerID="aef580658217fa9e7afb2f3b3555c062a4d1e1efa85b68e99b8abde0bb95d4c1"
Nov 03 19:51:46.737088 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:46.737066    5237 scope.go:122] "RemoveContainer" containerID="aef580658217fa9e7afb2f3b3555c062a4d1e1efa85b68e99b8abde0bb95d4c1"
Nov 03 19:51:46.737414 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:51:46.737389    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef580658217fa9e7afb2f3b3555c062a4d1e1efa85b68e99b8abde0bb95d4c1\": container with ID starting with aef580658217fa9e7afb2f3b3555c062a4d1e1efa85b68e99b8abde0bb95d4c1 not found: ID does not exist" containerID="aef580658217fa9e7afb2f3b3555c062a4d1e1efa85b68e99b8abde0bb95d4c1"
Nov 03 19:51:46.737493 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:46.737430    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef580658217fa9e7afb2f3b3555c062a4d1e1efa85b68e99b8abde0bb95d4c1"} err="failed to get container status \"aef580658217fa9e7afb2f3b3555c062a4d1e1efa85b68e99b8abde0bb95d4c1\": rpc error: code = NotFound desc = could not find container \"aef580658217fa9e7afb2f3b3555c062a4d1e1efa85b68e99b8abde0bb95d4c1\": container with ID starting with aef580658217fa9e7afb2f3b3555c062a4d1e1efa85b68e99b8abde0bb95d4c1 not found: ID does not exist"
Nov 03 19:51:46.754955 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:46.754922    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"]
Nov 03 19:51:46.759246 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:46.759217    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c54f85-665546547c-6zlrr"]
Nov 03 19:51:47.007748 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:47.007646    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:47.654423 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:47.654380    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:47.863296 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:47.863265    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af9b6d3c-e4c9-478c-9d19-36444a7c4e77" path="/var/lib/kubelet/pods/af9b6d3c-e4c9-478c-9d19-36444a7c4e77/volumes"
Nov 03 19:51:47.893315 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:47.893265    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:48.083195 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:48.083160    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:48.565515 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:48.565480    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:49.272860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:49.267918    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:49.685800 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:49.685672    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"]
Nov 03 19:51:49.686348 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:49.686291    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz" podUID="9145fea2-1fed-40d6-82e4-2f79350f9d8d" containerName="maas-api" containerID="cri-o://34ec9b321753646cd6a71ed518e1662a6544b7031fc7eabf7ff2966676b9ca41" gracePeriod=30
Nov 03 19:51:49.779435 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:49.779396    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:50.079883 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.057327    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:50.262656 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.262623    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:50.268676 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.268648    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:51:50.322867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.322809    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9145fea2-1fed-40d6-82e4-2f79350f9d8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9145fea2-1fed-40d6-82e4-2f79350f9d8d-maas-api-tls\") pod \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\" (UID: \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\") "
Nov 03 19:51:50.323260 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.322971    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9145fea2-1fed-40d6-82e4-2f79350f9d8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9145fea2-1fed-40d6-82e4-2f79350f9d8d-openshift-service-ca\") pod \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\" (UID: \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\") "
Nov 03 19:51:50.323260 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.323026    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9145fea2-1fed-40d6-82e4-2f79350f9d8d-kube-api-access-zdwv2\" (UniqueName: \"kubernetes.io/projected/9145fea2-1fed-40d6-82e4-2f79350f9d8d-kube-api-access-zdwv2\") pod \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\" (UID: \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\") "
Nov 03 19:51:50.323385 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.323354    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9145fea2-1fed-40d6-82e4-2f79350f9d8d-openshift-service-ca" pod "9145fea2-1fed-40d6-82e4-2f79350f9d8d" (UID: "9145fea2-1fed-40d6-82e4-2f79350f9d8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:50.323605 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.323587    5237 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9145fea2-1fed-40d6-82e4-2f79350f9d8d-openshift-service-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:50.325086 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.325055    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9145fea2-1fed-40d6-82e4-2f79350f9d8d-kube-api-access-zdwv2" pod "9145fea2-1fed-40d6-82e4-2f79350f9d8d" (UID: "9145fea2-1fed-40d6-82e4-2f79350f9d8d"). InnerVolumeSpecName "kube-api-access-zdwv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:50.325185 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.325147    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9145fea2-1fed-40d6-82e4-2f79350f9d8d-maas-api-tls" pod "9145fea2-1fed-40d6-82e4-2f79350f9d8d" (UID: "9145fea2-1fed-40d6-82e4-2f79350f9d8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:51:50.429864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.428513    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9145fea2-1fed-40d6-82e4-2f79350f9d8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9145fea2-1fed-40d6-82e4-2f79350f9d8d-metrics-certs\") pod \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\" (UID: \"9145fea2-1fed-40d6-82e4-2f79350f9d8d\") "
Nov 03 19:51:50.429864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.429406    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdwv2\" (UniqueName: \"kubernetes.io/projected/9145fea2-1fed-40d6-82e4-2f79350f9d8d-kube-api-access-zdwv2\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:50.429864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.429434    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9145fea2-1fed-40d6-82e4-2f79350f9d8d-maas-api-tls\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:50.457893 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.449173    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9145fea2-1fed-40d6-82e4-2f79350f9d8d-metrics-certs" pod "9145fea2-1fed-40d6-82e4-2f79350f9d8d" (UID: "9145fea2-1fed-40d6-82e4-2f79350f9d8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:51:50.457893 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.449777    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:50.530943 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.530907    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9145fea2-1fed-40d6-82e4-2f79350f9d8d-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:50.778811 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.778776    5237 generic.go:356] "Generic (PLEG): container finished" podID="9145fea2-1fed-40d6-82e4-2f79350f9d8d" containerID="34ec9b321753646cd6a71ed518e1662a6544b7031fc7eabf7ff2966676b9ca41" exitCode=0
Nov 03 19:51:50.779036 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.778830    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"
Nov 03 19:51:50.779036 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.778862    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz" event={"ID":"9145fea2-1fed-40d6-82e4-2f79350f9d8d","Type":"ContainerDied","Data":"34ec9b321753646cd6a71ed518e1662a6544b7031fc7eabf7ff2966676b9ca41"}
Nov 03 19:51:50.779036 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.778990    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz" event={"ID":"9145fea2-1fed-40d6-82e4-2f79350f9d8d","Type":"ContainerDied","Data":"3141ef8f945c4d8d5494e85348516a76e3ba63634788e5cf35450aeea632d591"}
Nov 03 19:51:50.779036 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.779013    5237 scope.go:122] "RemoveContainer" containerID="34ec9b321753646cd6a71ed518e1662a6544b7031fc7eabf7ff2966676b9ca41"
Nov 03 19:51:50.794156 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.794128    5237 scope.go:122] "RemoveContainer" containerID="34ec9b321753646cd6a71ed518e1662a6544b7031fc7eabf7ff2966676b9ca41"
Nov 03 19:51:50.794571 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:51:50.794538    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34ec9b321753646cd6a71ed518e1662a6544b7031fc7eabf7ff2966676b9ca41\": container with ID starting with 34ec9b321753646cd6a71ed518e1662a6544b7031fc7eabf7ff2966676b9ca41 not found: ID does not exist" containerID="34ec9b321753646cd6a71ed518e1662a6544b7031fc7eabf7ff2966676b9ca41"
Nov 03 19:51:50.794671 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.794595    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ec9b321753646cd6a71ed518e1662a6544b7031fc7eabf7ff2966676b9ca41"} err="failed to get container status \"34ec9b321753646cd6a71ed518e1662a6544b7031fc7eabf7ff2966676b9ca41\": rpc error: code = NotFound desc = could not find container \"34ec9b321753646cd6a71ed518e1662a6544b7031fc7eabf7ff2966676b9ca41\": container with ID starting with 34ec9b321753646cd6a71ed518e1662a6544b7031fc7eabf7ff2966676b9ca41 not found: ID does not exist"
Nov 03 19:51:50.807914 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.807887    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"]
Nov 03 19:51:50.813491 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.813465    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c54f85-66475cf6cd-bfzvz"]
Nov 03 19:51:50.880702 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:50.880641    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:51.309392 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:51.309350    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:51.860948 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:51.860909    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9145fea2-1fed-40d6-82e4-2f79350f9d8d" path="/var/lib/kubelet/pods/9145fea2-1fed-40d6-82e4-2f79350f9d8d/volumes"
Nov 03 19:51:52.236878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:52.236763    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:53.016462 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:53.015469    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:53.786859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:53.783475    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:54.418872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:54.418810    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:54.547459 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:54.547422    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:54.668958 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:54.668865    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"]
Nov 03 19:51:54.902383 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:54.902226    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:55.041140 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:55.039606    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:55.173584 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:55.173545    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:55.392919 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:55.385304    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:56.302864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:56.287820    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:57.040756 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.040717    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:57.277055 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.277004    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" containerID="cri-o://63a30f37d3853b3f6ea1ffecaf303f318e3302fb1743b9fccc7d587f57320424" gracePeriod=45
Nov 03 19:51:57.305986 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.305950    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" containerID="cri-o://07c14163bfef70bcb86543ff904a547f0cb9e5c653447fbdce16778f7384e2ab" gracePeriod=45
Nov 03 19:51:57.311235 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.311190    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": dial tcp 10.133.0.40:8000: connect: connection refused"
Nov 03 19:51:57.437577 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.437260    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": dial tcp 10.133.0.50:8000: connect: connection refused"
Nov 03 19:51:57.592453 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.592425    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:51:57.609905 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.609878    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:51:57.718617 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.718582    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-model-cache\") pod \"af4841d0-e4fd-4544-8924-cc21f3737772\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") "
Nov 03 19:51:57.718816 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.718635    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af4841d0-e4fd-4544-8924-cc21f3737772-tls-certs\" (UniqueName: \"kubernetes.io/secret/af4841d0-e4fd-4544-8924-cc21f3737772-tls-certs\") pod \"af4841d0-e4fd-4544-8924-cc21f3737772\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") "
Nov 03 19:51:57.718816 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.718767    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-dshm\") pod \"af4841d0-e4fd-4544-8924-cc21f3737772\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") "
Nov 03 19:51:57.718816 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.718809    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af4841d0-e4fd-4544-8924-cc21f3737772-kube-api-access-kpgr4\" (UniqueName: \"kubernetes.io/projected/af4841d0-e4fd-4544-8924-cc21f3737772-kube-api-access-kpgr4\") pod \"af4841d0-e4fd-4544-8924-cc21f3737772\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") "
Nov 03 19:51:57.719069 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.718858    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-dshm\") pod \"cfb69380-947e-474e-974f-9e17065379cf\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") "
Nov 03 19:51:57.719069 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.718915    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb69380-947e-474e-974f-9e17065379cf-kube-api-access-qfwk5\" (UniqueName: \"kubernetes.io/projected/cfb69380-947e-474e-974f-9e17065379cf-kube-api-access-qfwk5\") pod \"cfb69380-947e-474e-974f-9e17065379cf\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") "
Nov 03 19:51:57.719069 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.718988    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-model-cache" pod "af4841d0-e4fd-4544-8924-cc21f3737772" (UID: "af4841d0-e4fd-4544-8924-cc21f3737772"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:57.719069 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.719043    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb69380-947e-474e-974f-9e17065379cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb69380-947e-474e-974f-9e17065379cf-tls-certs\") pod \"cfb69380-947e-474e-974f-9e17065379cf\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") "
Nov 03 19:51:57.719280 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.719080    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-home\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-home\") pod \"af4841d0-e4fd-4544-8924-cc21f3737772\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") "
Nov 03 19:51:57.719280 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.719168    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-tmp-dir\") pod \"af4841d0-e4fd-4544-8924-cc21f3737772\" (UID: \"af4841d0-e4fd-4544-8924-cc21f3737772\") "
Nov 03 19:51:57.719280 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.719201    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-model-cache\") pod \"cfb69380-947e-474e-974f-9e17065379cf\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") "
Nov 03 19:51:57.719280 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.719244    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-home\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-home\") pod \"cfb69380-947e-474e-974f-9e17065379cf\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") "
Nov 03 19:51:57.719480 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.719332    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-tmp-dir\") pod \"cfb69380-947e-474e-974f-9e17065379cf\" (UID: \"cfb69380-947e-474e-974f-9e17065379cf\") "
Nov 03 19:51:57.719480 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.719444    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-home" pod "af4841d0-e4fd-4544-8924-cc21f3737772" (UID: "af4841d0-e4fd-4544-8924-cc21f3737772"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:57.719713 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.719688    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-tmp-dir" pod "cfb69380-947e-474e-974f-9e17065379cf" (UID: "cfb69380-947e-474e-974f-9e17065379cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:57.719913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.719891    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-tmp-dir" pod "af4841d0-e4fd-4544-8924-cc21f3737772" (UID: "af4841d0-e4fd-4544-8924-cc21f3737772"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:57.720201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.720179    5237 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-home\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.720289 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.720211    5237 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-tmp-dir\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.720289 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.720225    5237 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-tmp-dir\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.720289 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.720241    5237 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-model-cache\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.720289 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.720158    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-model-cache" pod "cfb69380-947e-474e-974f-9e17065379cf" (UID: "cfb69380-947e-474e-974f-9e17065379cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:57.720528 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.720387    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-home" pod "cfb69380-947e-474e-974f-9e17065379cf" (UID: "cfb69380-947e-474e-974f-9e17065379cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:57.721869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.721783    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af4841d0-e4fd-4544-8924-cc21f3737772-tls-certs" pod "af4841d0-e4fd-4544-8924-cc21f3737772" (UID: "af4841d0-e4fd-4544-8924-cc21f3737772"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:51:57.721869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.721803    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-dshm" pod "af4841d0-e4fd-4544-8924-cc21f3737772" (UID: "af4841d0-e4fd-4544-8924-cc21f3737772"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:57.721869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.721809    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4841d0-e4fd-4544-8924-cc21f3737772-kube-api-access-kpgr4" pod "af4841d0-e4fd-4544-8924-cc21f3737772" (UID: "af4841d0-e4fd-4544-8924-cc21f3737772"). InnerVolumeSpecName "kube-api-access-kpgr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:57.721869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.721806    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb69380-947e-474e-974f-9e17065379cf-tls-certs" pod "cfb69380-947e-474e-974f-9e17065379cf" (UID: "cfb69380-947e-474e-974f-9e17065379cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:51:57.722049 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.721883    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb69380-947e-474e-974f-9e17065379cf-kube-api-access-qfwk5" pod "cfb69380-947e-474e-974f-9e17065379cf" (UID: "cfb69380-947e-474e-974f-9e17065379cf"). InnerVolumeSpecName "kube-api-access-qfwk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:57.722442 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.722423    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-dshm" pod "cfb69380-947e-474e-974f-9e17065379cf" (UID: "cfb69380-947e-474e-974f-9e17065379cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:57.733557 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.733528    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:57.827941 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.825709    5237 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4841d0-e4fd-4544-8924-cc21f3737772-dshm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.827941 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.825762    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpgr4\" (UniqueName: \"kubernetes.io/projected/af4841d0-e4fd-4544-8924-cc21f3737772-kube-api-access-kpgr4\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.827941 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.825780    5237 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-dshm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.827941 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.825795    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfwk5\" (UniqueName: \"kubernetes.io/projected/cfb69380-947e-474e-974f-9e17065379cf-kube-api-access-qfwk5\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.827941 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.825812    5237 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb69380-947e-474e-974f-9e17065379cf-tls-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.827941 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.825827    5237 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-model-cache\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.827941 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.825872    5237 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb69380-947e-474e-974f-9e17065379cf-home\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.827941 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.825885    5237 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af4841d0-e4fd-4544-8924-cc21f3737772-tls-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.879821 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:57.879788    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:58.022917 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.020931    5237 generic.go:356] "Generic (PLEG): container finished" podID="cfb69380-947e-474e-974f-9e17065379cf" containerID="63a30f37d3853b3f6ea1ffecaf303f318e3302fb1743b9fccc7d587f57320424" exitCode=0
Nov 03 19:51:58.022917 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.021185    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" event={"ID":"cfb69380-947e-474e-974f-9e17065379cf","Type":"ContainerDied","Data":"63a30f37d3853b3f6ea1ffecaf303f318e3302fb1743b9fccc7d587f57320424"}
Nov 03 19:51:58.022917 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.021234    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws" event={"ID":"cfb69380-947e-474e-974f-9e17065379cf","Type":"ContainerDied","Data":"706a69ff82e9a8a3fff55ef6e0ce9c8a402d1aee69fb6464b6c7f26d5f07f5a0"}
Nov 03 19:51:58.022917 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.021260    5237 scope.go:122] "RemoveContainer" containerID="63a30f37d3853b3f6ea1ffecaf303f318e3302fb1743b9fccc7d587f57320424"
Nov 03 19:51:58.022917 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.021420    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"
Nov 03 19:51:58.026865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.025287    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:58.036861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.034020    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"]
Nov 03 19:51:58.036861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.034263    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq" podUID="92e6fabb-7410-46bc-a67f-40d9116f02eb" containerName="istio-proxy" containerID="cri-o://ac47927c63e565e280a9d248a2bf479df86334a23d0cf5b7b0b63b62fd0e4378" gracePeriod=30
Nov 03 19:51:58.036861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.036702    5237 generic.go:356] "Generic (PLEG): container finished" podID="af4841d0-e4fd-4544-8924-cc21f3737772" containerID="07c14163bfef70bcb86543ff904a547f0cb9e5c653447fbdce16778f7384e2ab" exitCode=0
Nov 03 19:51:58.036861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.036823    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" event={"ID":"af4841d0-e4fd-4544-8924-cc21f3737772","Type":"ContainerDied","Data":"07c14163bfef70bcb86543ff904a547f0cb9e5c653447fbdce16778f7384e2ab"}
Nov 03 19:51:58.036861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.036870    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg" event={"ID":"af4841d0-e4fd-4544-8924-cc21f3737772","Type":"ContainerDied","Data":"45405b8e3b3dfe01e302f9437ba1f7a2a14cfd6d94263d49fea07e78580ccc9f"}
Nov 03 19:51:58.037319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.036954    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"
Nov 03 19:51:58.102001 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.096592    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"]
Nov 03 19:51:58.102001 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.097037    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb" podUID="5fdb8431-30cd-4023-94e9-47f32412e2e7" containerName="istio-proxy" containerID="cri-o://97e20ca505dac2ef06e549c4f26275d9a4ee6c1b6a38b14f35c32e193459ded1" gracePeriod=30
Nov 03 19:51:58.117172 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.117144    5237 scope.go:122] "RemoveContainer" containerID="63a30f37d3853b3f6ea1ffecaf303f318e3302fb1743b9fccc7d587f57320424"
Nov 03 19:51:58.119398 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:51:58.117677    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63a30f37d3853b3f6ea1ffecaf303f318e3302fb1743b9fccc7d587f57320424\": container with ID starting with 63a30f37d3853b3f6ea1ffecaf303f318e3302fb1743b9fccc7d587f57320424 not found: ID does not exist" containerID="63a30f37d3853b3f6ea1ffecaf303f318e3302fb1743b9fccc7d587f57320424"
Nov 03 19:51:58.119398 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.117716    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a30f37d3853b3f6ea1ffecaf303f318e3302fb1743b9fccc7d587f57320424"} err="failed to get container status \"63a30f37d3853b3f6ea1ffecaf303f318e3302fb1743b9fccc7d587f57320424\": rpc error: code = NotFound desc = could not find container \"63a30f37d3853b3f6ea1ffecaf303f318e3302fb1743b9fccc7d587f57320424\": container with ID starting with 63a30f37d3853b3f6ea1ffecaf303f318e3302fb1743b9fccc7d587f57320424 not found: ID does not exist"
Nov 03 19:51:58.119398 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.117742    5237 scope.go:122] "RemoveContainer" containerID="07c14163bfef70bcb86543ff904a547f0cb9e5c653447fbdce16778f7384e2ab"
Nov 03 19:51:58.132390 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.131745    5237 scope.go:122] "RemoveContainer" containerID="07c14163bfef70bcb86543ff904a547f0cb9e5c653447fbdce16778f7384e2ab"
Nov 03 19:51:58.132774 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:51:58.132744    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c14163bfef70bcb86543ff904a547f0cb9e5c653447fbdce16778f7384e2ab\": container with ID starting with 07c14163bfef70bcb86543ff904a547f0cb9e5c653447fbdce16778f7384e2ab not found: ID does not exist" containerID="07c14163bfef70bcb86543ff904a547f0cb9e5c653447fbdce16778f7384e2ab"
Nov 03 19:51:58.132955 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.132782    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c14163bfef70bcb86543ff904a547f0cb9e5c653447fbdce16778f7384e2ab"} err="failed to get container status \"07c14163bfef70bcb86543ff904a547f0cb9e5c653447fbdce16778f7384e2ab\": rpc error: code = NotFound desc = could not find container \"07c14163bfef70bcb86543ff904a547f0cb9e5c653447fbdce16778f7384e2ab\": container with ID starting with 07c14163bfef70bcb86543ff904a547f0cb9e5c653447fbdce16778f7384e2ab not found: ID does not exist"
Nov 03 19:51:58.133039 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.132986    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"]
Nov 03 19:51:58.136487 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.136466    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/distinct-2-w1-2c3fc4-kserve-74dbf55ddb-cbgws"]
Nov 03 19:51:58.155505 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.155469    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"]
Nov 03 19:51:58.158459 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.158434    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/premium-w1-2c3fc4-kserve-684d46456b-8pcvg"]
Nov 03 19:51:58.241870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.241711    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:58.938861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:58.937585    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:59.467823 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:59.467782    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:59.633867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:59.623388    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:59.675857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:59.674936    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97" podUID="adf802a7-4ee5-439e-8748-8d390c5f1f51" containerName="payload-pre-processing" containerID="cri-o://74eb9d2e40a7eb0bcbc1c769d1caffa88a23fb3f8f2e4816090955bca0c502c3" gracePeriod=25
Nov 03 19:51:59.766865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:59.765472    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:59.939865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:59.936715    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" path="/var/lib/kubelet/pods/af4841d0-e4fd-4544-8924-cc21f3737772/volumes"
Nov 03 19:51:59.939865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:51:59.937455    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb69380-947e-474e-974f-9e17065379cf" path="/var/lib/kubelet/pods/cfb69380-947e-474e-974f-9e17065379cf/volumes"
Nov 03 19:52:00.044069 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:00.044035    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:00.133903 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:00.133146    5237 generic.go:356] "Generic (PLEG): container finished" podID="adf802a7-4ee5-439e-8748-8d390c5f1f51" containerID="74eb9d2e40a7eb0bcbc1c769d1caffa88a23fb3f8f2e4816090955bca0c502c3" exitCode=0
Nov 03 19:52:00.133903 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:00.133393    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97" event={"ID":"adf802a7-4ee5-439e-8748-8d390c5f1f51","Type":"ContainerDied","Data":"74eb9d2e40a7eb0bcbc1c769d1caffa88a23fb3f8f2e4816090955bca0c502c3"}
Nov 03 19:52:00.170967 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:00.170939    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"
Nov 03 19:52:00.229478 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:00.229444    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf802a7-4ee5-439e-8748-8d390c5f1f51-kube-api-access-pdbdm\" (UniqueName: \"kubernetes.io/projected/adf802a7-4ee5-439e-8748-8d390c5f1f51-kube-api-access-pdbdm\") pod \"adf802a7-4ee5-439e-8748-8d390c5f1f51\" (UID: \"adf802a7-4ee5-439e-8748-8d390c5f1f51\") "
Nov 03 19:52:00.229662 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:00.229596    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adf802a7-4ee5-439e-8748-8d390c5f1f51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf802a7-4ee5-439e-8748-8d390c5f1f51-plugins-config-volume\") pod \"adf802a7-4ee5-439e-8748-8d390c5f1f51\" (UID: \"adf802a7-4ee5-439e-8748-8d390c5f1f51\") "
Nov 03 19:52:00.230067 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:00.230034    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf802a7-4ee5-439e-8748-8d390c5f1f51-plugins-config-volume" pod "adf802a7-4ee5-439e-8748-8d390c5f1f51" (UID: "adf802a7-4ee5-439e-8748-8d390c5f1f51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:00.230357 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:00.230331    5237 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf802a7-4ee5-439e-8748-8d390c5f1f51-plugins-config-volume\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:00.232359 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:00.232320    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf802a7-4ee5-439e-8748-8d390c5f1f51-kube-api-access-pdbdm" pod "adf802a7-4ee5-439e-8748-8d390c5f1f51" (UID: "adf802a7-4ee5-439e-8748-8d390c5f1f51"). InnerVolumeSpecName "kube-api-access-pdbdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:00.332023 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:00.331943    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdbdm\" (UniqueName: \"kubernetes.io/projected/adf802a7-4ee5-439e-8748-8d390c5f1f51-kube-api-access-pdbdm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:00.480507 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:00.480459    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:00.859857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:00.848600    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:01.144672 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:01.144586    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97" event={"ID":"adf802a7-4ee5-439e-8748-8d390c5f1f51","Type":"ContainerDied","Data":"f06f1fbc939f17210784424f271d642701c6902486f3413bade7eb55d9458e23"}
Nov 03 19:52:01.144672 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:01.144625    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"
Nov 03 19:52:01.144672 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:01.144651    5237 scope.go:122] "RemoveContainer" containerID="74eb9d2e40a7eb0bcbc1c769d1caffa88a23fb3f8f2e4816090955bca0c502c3"
Nov 03 19:52:01.174196 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:01.174161    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"]
Nov 03 19:52:01.181770 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:01.181745    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-29ab98-78fb8cc5d7-nqp97"]
Nov 03 19:52:01.450569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:01.450484    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:01.862631 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:01.862597    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adf802a7-4ee5-439e-8748-8d390c5f1f51" path="/var/lib/kubelet/pods/adf802a7-4ee5-439e-8748-8d390c5f1f51/volumes"
Nov 03 19:52:02.430631 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:02.430578    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:03.193623 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.193587    5237 generic.go:356] "Generic (PLEG): container finished" podID="92e6fabb-7410-46bc-a67f-40d9116f02eb" containerID="ac47927c63e565e280a9d248a2bf479df86334a23d0cf5b7b0b63b62fd0e4378" exitCode=0
Nov 03 19:52:03.193793 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.193654    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq" event={"ID":"92e6fabb-7410-46bc-a67f-40d9116f02eb","Type":"ContainerDied","Data":"ac47927c63e565e280a9d248a2bf479df86334a23d0cf5b7b0b63b62fd0e4378"}
Nov 03 19:52:03.225178 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.225148    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:03.513427 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.513399    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:52:03.577161 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.577125    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-credential-socket\") pod \"5fdb8431-30cd-4023-94e9-47f32412e2e7\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") "
Nov 03 19:52:03.577161 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.577164    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-workload-socket\") pod \"5fdb8431-30cd-4023-94e9-47f32412e2e7\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") "
Nov 03 19:52:03.577426 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.577197    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-workload-certs\") pod \"5fdb8431-30cd-4023-94e9-47f32412e2e7\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") "
Nov 03 19:52:03.577426 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.577230    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fdb8431-30cd-4023-94e9-47f32412e2e7-kube-api-access-vnpzf\" (UniqueName: \"kubernetes.io/projected/5fdb8431-30cd-4023-94e9-47f32412e2e7-kube-api-access-vnpzf\") pod \"5fdb8431-30cd-4023-94e9-47f32412e2e7\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") "
Nov 03 19:52:03.577426 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.577282    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fdb8431-30cd-4023-94e9-47f32412e2e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fdb8431-30cd-4023-94e9-47f32412e2e7-istiod-ca-cert\") pod \"5fdb8431-30cd-4023-94e9-47f32412e2e7\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") "
Nov 03 19:52:03.577426 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.577332    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-token\" (UniqueName: \"kubernetes.io/projected/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-token\") pod \"5fdb8431-30cd-4023-94e9-47f32412e2e7\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") "
Nov 03 19:52:03.577426 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.577363    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-envoy\") pod \"5fdb8431-30cd-4023-94e9-47f32412e2e7\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") "
Nov 03 19:52:03.577426 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.577412    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-credential-socket" pod "5fdb8431-30cd-4023-94e9-47f32412e2e7" (UID: "5fdb8431-30cd-4023-94e9-47f32412e2e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:03.577791 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.577438    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-podinfo\") pod \"5fdb8431-30cd-4023-94e9-47f32412e2e7\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") "
Nov 03 19:52:03.577791 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.577483    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-data\") pod \"5fdb8431-30cd-4023-94e9-47f32412e2e7\" (UID: \"5fdb8431-30cd-4023-94e9-47f32412e2e7\") "
Nov 03 19:52:03.577791 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.577547    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-workload-socket" pod "5fdb8431-30cd-4023-94e9-47f32412e2e7" (UID: "5fdb8431-30cd-4023-94e9-47f32412e2e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:03.577791 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.577566    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-workload-certs" pod "5fdb8431-30cd-4023-94e9-47f32412e2e7" (UID: "5fdb8431-30cd-4023-94e9-47f32412e2e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:03.577791 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.577709    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fdb8431-30cd-4023-94e9-47f32412e2e7-istiod-ca-cert" pod "5fdb8431-30cd-4023-94e9-47f32412e2e7" (UID: "5fdb8431-30cd-4023-94e9-47f32412e2e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:03.578069 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.577928    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-data" pod "5fdb8431-30cd-4023-94e9-47f32412e2e7" (UID: "5fdb8431-30cd-4023-94e9-47f32412e2e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:03.578544 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.578519    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.578660 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.578548    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.578660 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.578563    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.578660 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.578579    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fdb8431-30cd-4023-94e9-47f32412e2e7-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.578660 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.578594    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.579726 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.579689    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdb8431-30cd-4023-94e9-47f32412e2e7-kube-api-access-vnpzf" pod "5fdb8431-30cd-4023-94e9-47f32412e2e7" (UID: "5fdb8431-30cd-4023-94e9-47f32412e2e7"). InnerVolumeSpecName "kube-api-access-vnpzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:03.579974 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.579948    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-envoy" pod "5fdb8431-30cd-4023-94e9-47f32412e2e7" (UID: "5fdb8431-30cd-4023-94e9-47f32412e2e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:03.580096 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.579951    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-token" pod "5fdb8431-30cd-4023-94e9-47f32412e2e7" (UID: "5fdb8431-30cd-4023-94e9-47f32412e2e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:03.580177 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.580155    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-podinfo" pod "5fdb8431-30cd-4023-94e9-47f32412e2e7" (UID: "5fdb8431-30cd-4023-94e9-47f32412e2e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:52:03.681339 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.681303    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnpzf\" (UniqueName: \"kubernetes.io/projected/5fdb8431-30cd-4023-94e9-47f32412e2e7-kube-api-access-vnpzf\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.681339 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.681341    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.681563 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.681359    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.681563 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.681373    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fdb8431-30cd-4023-94e9-47f32412e2e7-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.748885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.748776    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:52:03.782039 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.782000    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e6fabb-7410-46bc-a67f-40d9116f02eb-kube-api-access-zr6tk\" (UniqueName: \"kubernetes.io/projected/92e6fabb-7410-46bc-a67f-40d9116f02eb-kube-api-access-zr6tk\") pod \"92e6fabb-7410-46bc-a67f-40d9116f02eb\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") "
Nov 03 19:52:03.782256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.782073    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-credential-socket\") pod \"92e6fabb-7410-46bc-a67f-40d9116f02eb\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") "
Nov 03 19:52:03.782256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.782111    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e6fabb-7410-46bc-a67f-40d9116f02eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e6fabb-7410-46bc-a67f-40d9116f02eb-istiod-ca-cert\") pod \"92e6fabb-7410-46bc-a67f-40d9116f02eb\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") "
Nov 03 19:52:03.782256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.782175    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-workload-socket\") pod \"92e6fabb-7410-46bc-a67f-40d9116f02eb\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") "
Nov 03 19:52:03.782256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.782254    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-token\" (UniqueName: \"kubernetes.io/projected/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-token\") pod \"92e6fabb-7410-46bc-a67f-40d9116f02eb\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") "
Nov 03 19:52:03.782490 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.782293    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-envoy\") pod \"92e6fabb-7410-46bc-a67f-40d9116f02eb\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") "
Nov 03 19:52:03.782490 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.782388    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-podinfo\") pod \"92e6fabb-7410-46bc-a67f-40d9116f02eb\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") "
Nov 03 19:52:03.782490 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.782432    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-data\") pod \"92e6fabb-7410-46bc-a67f-40d9116f02eb\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") "
Nov 03 19:52:03.782490 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.782471    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-workload-certs\") pod \"92e6fabb-7410-46bc-a67f-40d9116f02eb\" (UID: \"92e6fabb-7410-46bc-a67f-40d9116f02eb\") "
Nov 03 19:52:03.802899 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.793013    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-workload-socket" pod "92e6fabb-7410-46bc-a67f-40d9116f02eb" (UID: "92e6fabb-7410-46bc-a67f-40d9116f02eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:03.802899 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.793387    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-workload-certs" pod "92e6fabb-7410-46bc-a67f-40d9116f02eb" (UID: "92e6fabb-7410-46bc-a67f-40d9116f02eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:03.802899 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.793724    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-credential-socket" pod "92e6fabb-7410-46bc-a67f-40d9116f02eb" (UID: "92e6fabb-7410-46bc-a67f-40d9116f02eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:03.802899 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.802484    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e6fabb-7410-46bc-a67f-40d9116f02eb-istiod-ca-cert" pod "92e6fabb-7410-46bc-a67f-40d9116f02eb" (UID: "92e6fabb-7410-46bc-a67f-40d9116f02eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:03.802899 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.802748    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-data" pod "92e6fabb-7410-46bc-a67f-40d9116f02eb" (UID: "92e6fabb-7410-46bc-a67f-40d9116f02eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:03.808726 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.808692    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-podinfo" pod "92e6fabb-7410-46bc-a67f-40d9116f02eb" (UID: "92e6fabb-7410-46bc-a67f-40d9116f02eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:52:03.810365 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.810241    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-envoy" pod "92e6fabb-7410-46bc-a67f-40d9116f02eb" (UID: "92e6fabb-7410-46bc-a67f-40d9116f02eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:03.810507 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.810394    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e6fabb-7410-46bc-a67f-40d9116f02eb-kube-api-access-zr6tk" pod "92e6fabb-7410-46bc-a67f-40d9116f02eb" (UID: "92e6fabb-7410-46bc-a67f-40d9116f02eb"). InnerVolumeSpecName "kube-api-access-zr6tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:03.811006 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.810979    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-token" pod "92e6fabb-7410-46bc-a67f-40d9116f02eb" (UID: "92e6fabb-7410-46bc-a67f-40d9116f02eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:03.852905 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.852338    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:03.892882 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.892851    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.892882 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.892877    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.892882 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.892887    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.893181 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.892895    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.893181 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.892903    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.893181 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.892914    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr6tk\" (UniqueName: \"kubernetes.io/projected/92e6fabb-7410-46bc-a67f-40d9116f02eb-kube-api-access-zr6tk\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.893181 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.892929    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.893181 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.892938    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e6fabb-7410-46bc-a67f-40d9116f02eb-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.893181 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.892947    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6fabb-7410-46bc-a67f-40d9116f02eb-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.988562 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:03.988524    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:04.158346 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:04.156072    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:04.205151 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:04.205120    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"
Nov 03 19:52:04.205352 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:04.205157    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq" event={"ID":"92e6fabb-7410-46bc-a67f-40d9116f02eb","Type":"ContainerDied","Data":"92e1385a647a15006c394d92d2bba135fd72e1f86881c23e9e25efed5959a1b0"}
Nov 03 19:52:04.205352 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:04.205215    5237 scope.go:122] "RemoveContainer" containerID="ac47927c63e565e280a9d248a2bf479df86334a23d0cf5b7b0b63b62fd0e4378"
Nov 03 19:52:04.206907 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:04.206880    5237 generic.go:356] "Generic (PLEG): container finished" podID="5fdb8431-30cd-4023-94e9-47f32412e2e7" containerID="97e20ca505dac2ef06e549c4f26275d9a4ee6c1b6a38b14f35c32e193459ded1" exitCode=0
Nov 03 19:52:04.207052 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:04.206935    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb" event={"ID":"5fdb8431-30cd-4023-94e9-47f32412e2e7","Type":"ContainerDied","Data":"97e20ca505dac2ef06e549c4f26275d9a4ee6c1b6a38b14f35c32e193459ded1"}
Nov 03 19:52:04.207052 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:04.206961    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"
Nov 03 19:52:04.207052 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:04.206979    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb" event={"ID":"5fdb8431-30cd-4023-94e9-47f32412e2e7","Type":"ContainerDied","Data":"c2a2f11f08067eb422df67de64971d4c36124743417f72fd91db7fa7d79a716f"}
Nov 03 19:52:04.215912 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:04.215887    5237 scope.go:122] "RemoveContainer" containerID="97e20ca505dac2ef06e549c4f26275d9a4ee6c1b6a38b14f35c32e193459ded1"
Nov 03 19:52:04.225207 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:04.225044    5237 scope.go:122] "RemoveContainer" containerID="97e20ca505dac2ef06e549c4f26275d9a4ee6c1b6a38b14f35c32e193459ded1"
Nov 03 19:52:04.225372 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:52:04.225351    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e20ca505dac2ef06e549c4f26275d9a4ee6c1b6a38b14f35c32e193459ded1\": container with ID starting with 97e20ca505dac2ef06e549c4f26275d9a4ee6c1b6a38b14f35c32e193459ded1 not found: ID does not exist" containerID="97e20ca505dac2ef06e549c4f26275d9a4ee6c1b6a38b14f35c32e193459ded1"
Nov 03 19:52:04.225413 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:04.225381    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e20ca505dac2ef06e549c4f26275d9a4ee6c1b6a38b14f35c32e193459ded1"} err="failed to get container status \"97e20ca505dac2ef06e549c4f26275d9a4ee6c1b6a38b14f35c32e193459ded1\": rpc error: code = NotFound desc = could not find container \"97e20ca505dac2ef06e549c4f26275d9a4ee6c1b6a38b14f35c32e193459ded1\": container with ID starting with 97e20ca505dac2ef06e549c4f26275d9a4ee6c1b6a38b14f35c32e193459ded1 not found: ID does not exist"
Nov 03 19:52:04.269635 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:04.269601    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"]
Nov 03 19:52:04.279246 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:04.279219    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tk2cq"]
Nov 03 19:52:04.372250 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:04.372170    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"]
Nov 03 19:52:04.396910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:04.396872    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-nzldb"]
Nov 03 19:52:04.559918 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:04.559874    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:05.034881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:05.034377    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:05.381361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:05.381272    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:05.833646 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:05.833610    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:05.891990 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:05.886154    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fdb8431-30cd-4023-94e9-47f32412e2e7" path="/var/lib/kubelet/pods/5fdb8431-30cd-4023-94e9-47f32412e2e7/volumes"
Nov 03 19:52:05.891990 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:05.886946    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e6fabb-7410-46bc-a67f-40d9116f02eb" path="/var/lib/kubelet/pods/92e6fabb-7410-46bc-a67f-40d9116f02eb/volumes"
Nov 03 19:52:06.477105 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:06.477069    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:06.631712 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:06.631661    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:06.800878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:06.800727    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:07.183149 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:07.183041    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:07.326479 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:07.326414    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:07.850730 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:07.850694    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:08.347245 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:08.347203    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:08.483385 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:08.483344    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:08.659314 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:08.659002    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:08.810522 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:08.810486    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:08.874034 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:08.873998    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"]
Nov 03 19:52:08.899364 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:08.899325    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"]
Nov 03 19:52:09.017085 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:09.017002    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:09.210968 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:09.207255    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:10.078855 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:10.074887    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:10.839955 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:10.839915    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:11.669326 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:11.669167    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:11.867858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:11.867797    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:12.026206 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:12.026116    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:12.173773 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:12.173738    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:12.619667 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:12.619627    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:13.155435 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:13.155377    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:13.256958 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:52:13.256924    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:52:13.874380 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:13.874334    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p" podUID="84914ee8-d996-409e-a212-d29119cc5de8" containerName="payload-processing" containerID="cri-o://9f95679c81307d8103509a7e97ff0a493d8d1b9dd8efadbfbd02177cdf43a8a9" gracePeriod=25
Nov 03 19:52:13.900393 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:13.900344    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd" podUID="fe7148f3-a2df-43bb-bf04-d5a0fddc1a13" containerName="payload-pre-processing" containerID="cri-o://d8b770215c431b8c26e1e26cdd5842f6b8eac40fe94e42fd83844e6de351a13a" gracePeriod=25
Nov 03 19:52:14.197060 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.195601    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:14.235932 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.235539    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"
Nov 03 19:52:14.320352 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.320319    5237 generic.go:356] "Generic (PLEG): container finished" podID="fe7148f3-a2df-43bb-bf04-d5a0fddc1a13" containerID="d8b770215c431b8c26e1e26cdd5842f6b8eac40fe94e42fd83844e6de351a13a" exitCode=0
Nov 03 19:52:14.320531 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.320392    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd" event={"ID":"fe7148f3-a2df-43bb-bf04-d5a0fddc1a13","Type":"ContainerDied","Data":"d8b770215c431b8c26e1e26cdd5842f6b8eac40fe94e42fd83844e6de351a13a"}
Nov 03 19:52:14.321829 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.321803    5237 generic.go:356] "Generic (PLEG): container finished" podID="84914ee8-d996-409e-a212-d29119cc5de8" containerID="9f95679c81307d8103509a7e97ff0a493d8d1b9dd8efadbfbd02177cdf43a8a9" exitCode=0
Nov 03 19:52:14.321983 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.321938    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p" event={"ID":"84914ee8-d996-409e-a212-d29119cc5de8","Type":"ContainerDied","Data":"9f95679c81307d8103509a7e97ff0a493d8d1b9dd8efadbfbd02177cdf43a8a9"}
Nov 03 19:52:14.321983 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.321975    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p" event={"ID":"84914ee8-d996-409e-a212-d29119cc5de8","Type":"ContainerDied","Data":"e1fe30b5e857cb0db8e32005cf5c9ab5356fc38fad986cb05299fc84936ae8c3"}
Nov 03 19:52:14.322064 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.321997    5237 scope.go:122] "RemoveContainer" containerID="9f95679c81307d8103509a7e97ff0a493d8d1b9dd8efadbfbd02177cdf43a8a9"
Nov 03 19:52:14.322143 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.322117    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"
Nov 03 19:52:14.332090 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.332073    5237 scope.go:122] "RemoveContainer" containerID="9f95679c81307d8103509a7e97ff0a493d8d1b9dd8efadbfbd02177cdf43a8a9"
Nov 03 19:52:14.332404 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:52:14.332370    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f95679c81307d8103509a7e97ff0a493d8d1b9dd8efadbfbd02177cdf43a8a9\": container with ID starting with 9f95679c81307d8103509a7e97ff0a493d8d1b9dd8efadbfbd02177cdf43a8a9 not found: ID does not exist" containerID="9f95679c81307d8103509a7e97ff0a493d8d1b9dd8efadbfbd02177cdf43a8a9"
Nov 03 19:52:14.332499 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.332401    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f95679c81307d8103509a7e97ff0a493d8d1b9dd8efadbfbd02177cdf43a8a9"} err="failed to get container status \"9f95679c81307d8103509a7e97ff0a493d8d1b9dd8efadbfbd02177cdf43a8a9\": rpc error: code = NotFound desc = could not find container \"9f95679c81307d8103509a7e97ff0a493d8d1b9dd8efadbfbd02177cdf43a8a9\": container with ID starting with 9f95679c81307d8103509a7e97ff0a493d8d1b9dd8efadbfbd02177cdf43a8a9 not found: ID does not exist"
Nov 03 19:52:14.356528 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.356497    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:14.379867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.379826    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"
Nov 03 19:52:14.413172 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.413136    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe7148f3-a2df-43bb-bf04-d5a0fddc1a13-kube-api-access-fwqkw\" (UniqueName: \"kubernetes.io/projected/fe7148f3-a2df-43bb-bf04-d5a0fddc1a13-kube-api-access-fwqkw\") pod \"fe7148f3-a2df-43bb-bf04-d5a0fddc1a13\" (UID: \"fe7148f3-a2df-43bb-bf04-d5a0fddc1a13\") "
Nov 03 19:52:14.413440 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.413276    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe7148f3-a2df-43bb-bf04-d5a0fddc1a13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7148f3-a2df-43bb-bf04-d5a0fddc1a13-plugins-config-volume\") pod \"fe7148f3-a2df-43bb-bf04-d5a0fddc1a13\" (UID: \"fe7148f3-a2df-43bb-bf04-d5a0fddc1a13\") "
Nov 03 19:52:14.413440 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.413314    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84914ee8-d996-409e-a212-d29119cc5de8-kube-api-access-4prd7\" (UniqueName: \"kubernetes.io/projected/84914ee8-d996-409e-a212-d29119cc5de8-kube-api-access-4prd7\") pod \"84914ee8-d996-409e-a212-d29119cc5de8\" (UID: \"84914ee8-d996-409e-a212-d29119cc5de8\") "
Nov 03 19:52:14.413440 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.413390    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84914ee8-d996-409e-a212-d29119cc5de8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84914ee8-d996-409e-a212-d29119cc5de8-plugins-config-volume\") pod \"84914ee8-d996-409e-a212-d29119cc5de8\" (UID: \"84914ee8-d996-409e-a212-d29119cc5de8\") "
Nov 03 19:52:14.414373 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.414335    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe7148f3-a2df-43bb-bf04-d5a0fddc1a13-plugins-config-volume" pod "fe7148f3-a2df-43bb-bf04-d5a0fddc1a13" (UID: "fe7148f3-a2df-43bb-bf04-d5a0fddc1a13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:14.414505 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.414471    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84914ee8-d996-409e-a212-d29119cc5de8-plugins-config-volume" pod "84914ee8-d996-409e-a212-d29119cc5de8" (UID: "84914ee8-d996-409e-a212-d29119cc5de8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:14.415961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.415922    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe7148f3-a2df-43bb-bf04-d5a0fddc1a13-kube-api-access-fwqkw" pod "fe7148f3-a2df-43bb-bf04-d5a0fddc1a13" (UID: "fe7148f3-a2df-43bb-bf04-d5a0fddc1a13"). InnerVolumeSpecName "kube-api-access-fwqkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:14.416375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.416351    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84914ee8-d996-409e-a212-d29119cc5de8-kube-api-access-4prd7" pod "84914ee8-d996-409e-a212-d29119cc5de8" (UID: "84914ee8-d996-409e-a212-d29119cc5de8"). InnerVolumeSpecName "kube-api-access-4prd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:14.514976 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.514879    5237 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7148f3-a2df-43bb-bf04-d5a0fddc1a13-plugins-config-volume\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:14.514976 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.514910    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4prd7\" (UniqueName: \"kubernetes.io/projected/84914ee8-d996-409e-a212-d29119cc5de8-kube-api-access-4prd7\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:14.514976 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.514921    5237 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84914ee8-d996-409e-a212-d29119cc5de8-plugins-config-volume\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:14.514976 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.514931    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwqkw\" (UniqueName: \"kubernetes.io/projected/fe7148f3-a2df-43bb-bf04-d5a0fddc1a13-kube-api-access-fwqkw\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:14.653878 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.653827    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"]
Nov 03 19:52:14.659200 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:14.659171    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2b2e60-97fc74b5f-lg24p"]
Nov 03 19:52:15.197192 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:15.197101    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:15.330687 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:15.330658    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"
Nov 03 19:52:15.330881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:15.330652    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd" event={"ID":"fe7148f3-a2df-43bb-bf04-d5a0fddc1a13","Type":"ContainerDied","Data":"61ed9f1946f95ce07b7e175fc173d7bc007bacf14c135d7351540c2f34868d7c"}
Nov 03 19:52:15.330881 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:15.330790    5237 scope.go:122] "RemoveContainer" containerID="d8b770215c431b8c26e1e26cdd5842f6b8eac40fe94e42fd83844e6de351a13a"
Nov 03 19:52:15.356943 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:15.356909    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"]
Nov 03 19:52:15.362678 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:15.362646    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b2e60-855b47db8c-xvnsd"]
Nov 03 19:52:15.847259 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:15.847220    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:15.919543 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:15.911309    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84914ee8-d996-409e-a212-d29119cc5de8" path="/var/lib/kubelet/pods/84914ee8-d996-409e-a212-d29119cc5de8/volumes"
Nov 03 19:52:15.919543 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:15.915011    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe7148f3-a2df-43bb-bf04-d5a0fddc1a13" path="/var/lib/kubelet/pods/fe7148f3-a2df-43bb-bf04-d5a0fddc1a13/volumes"
Nov 03 19:52:16.072975 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:16.072941    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:16.228338 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:16.228214    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:16.392559 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:16.392523    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"]
Nov 03 19:52:16.392803 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:16.392777    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerName="istio-proxy" containerID="cri-o://78334ef3ed948b6a5413211648331cc6760a29bcb35e8e0645bb59022ff64c08" gracePeriod=30
Nov 03 19:52:16.396482 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:16.396278    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"]
Nov 03 19:52:16.396631 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:16.396546    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" podUID="e63a1c5a-826e-487e-b3eb-eab4278b8049" containerName="istio-proxy" containerID="cri-o://36a30a9ec5c8248309e48633fbf8924b7ec2d1ff29a8eb2e1b39c94a10fe2bc2" gracePeriod=30
Nov 03 19:52:16.400357 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:16.400332    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"]
Nov 03 19:52:16.400534 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:16.400510    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm" podUID="2b4a25ae-6965-4713-ab67-327317f678d8" containerName="istio-proxy" containerID="cri-o://7208ce730ab4cc3e006c968ed14b4def81f8c01b75fb3e5aad77dc714cd1b475" gracePeriod=30
Nov 03 19:52:16.470856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:16.468991    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:16.555603 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:16.555564    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"]
Nov 03 19:52:16.686757 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:16.686714    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg" podUID="17572c14-846c-4c5d-a141-6d36045da219" containerName="maas-api" containerID="cri-o://7124099ad916ab7128e4a00c8f16b9ee2c2f9666c198dc8c6749407c2f5d95c1" gracePeriod=30
Nov 03 19:52:16.692038 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:16.687442    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:17.013866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.012675    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:17.230350 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.230274    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:52:17.233794 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.233766    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17572c14-846c-4c5d-a141-6d36045da219-kube-api-access-76b97\" (UniqueName: \"kubernetes.io/projected/17572c14-846c-4c5d-a141-6d36045da219-kube-api-access-76b97\") pod \"17572c14-846c-4c5d-a141-6d36045da219\" (UID: \"17572c14-846c-4c5d-a141-6d36045da219\") "
Nov 03 19:52:17.233925 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.233868    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17572c14-846c-4c5d-a141-6d36045da219-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17572c14-846c-4c5d-a141-6d36045da219-maas-api-tls\") pod \"17572c14-846c-4c5d-a141-6d36045da219\" (UID: \"17572c14-846c-4c5d-a141-6d36045da219\") "
Nov 03 19:52:17.234054 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.234041    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17572c14-846c-4c5d-a141-6d36045da219-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17572c14-846c-4c5d-a141-6d36045da219-openshift-service-ca\") pod \"17572c14-846c-4c5d-a141-6d36045da219\" (UID: \"17572c14-846c-4c5d-a141-6d36045da219\") "
Nov 03 19:52:17.234512 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.234101    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17572c14-846c-4c5d-a141-6d36045da219-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17572c14-846c-4c5d-a141-6d36045da219-metrics-certs\") pod \"17572c14-846c-4c5d-a141-6d36045da219\" (UID: \"17572c14-846c-4c5d-a141-6d36045da219\") "
Nov 03 19:52:17.234512 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.234357    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17572c14-846c-4c5d-a141-6d36045da219-openshift-service-ca" pod "17572c14-846c-4c5d-a141-6d36045da219" (UID: "17572c14-846c-4c5d-a141-6d36045da219"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:17.234512 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.234498    5237 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17572c14-846c-4c5d-a141-6d36045da219-openshift-service-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:17.236394 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.236366    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17572c14-846c-4c5d-a141-6d36045da219-maas-api-tls" pod "17572c14-846c-4c5d-a141-6d36045da219" (UID: "17572c14-846c-4c5d-a141-6d36045da219"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:17.236504 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.236417    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17572c14-846c-4c5d-a141-6d36045da219-metrics-certs" pod "17572c14-846c-4c5d-a141-6d36045da219" (UID: "17572c14-846c-4c5d-a141-6d36045da219"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:17.236504 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.236443    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17572c14-846c-4c5d-a141-6d36045da219-kube-api-access-76b97" pod "17572c14-846c-4c5d-a141-6d36045da219" (UID: "17572c14-846c-4c5d-a141-6d36045da219"). InnerVolumeSpecName "kube-api-access-76b97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:17.301568 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.301534    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:17.335438 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.335392    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17572c14-846c-4c5d-a141-6d36045da219-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:17.335438 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.335431    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76b97\" (UniqueName: \"kubernetes.io/projected/17572c14-846c-4c5d-a141-6d36045da219-kube-api-access-76b97\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:17.335438 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.335446    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17572c14-846c-4c5d-a141-6d36045da219-maas-api-tls\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:17.354063 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.354026    5237 generic.go:356] "Generic (PLEG): container finished" podID="17572c14-846c-4c5d-a141-6d36045da219" containerID="7124099ad916ab7128e4a00c8f16b9ee2c2f9666c198dc8c6749407c2f5d95c1" exitCode=0
Nov 03 19:52:17.354222 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.354079    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"
Nov 03 19:52:17.354222 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.354109    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg" event={"ID":"17572c14-846c-4c5d-a141-6d36045da219","Type":"ContainerDied","Data":"7124099ad916ab7128e4a00c8f16b9ee2c2f9666c198dc8c6749407c2f5d95c1"}
Nov 03 19:52:17.354222 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.354170    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg" event={"ID":"17572c14-846c-4c5d-a141-6d36045da219","Type":"ContainerDied","Data":"e96fdfd6c9b32d446625e572861d6386439ae6c5fe1bdf94066bcde0e896ed83"}
Nov 03 19:52:17.354222 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.354194    5237 scope.go:122] "RemoveContainer" containerID="7124099ad916ab7128e4a00c8f16b9ee2c2f9666c198dc8c6749407c2f5d95c1"
Nov 03 19:52:17.368091 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.367992    5237 scope.go:122] "RemoveContainer" containerID="7124099ad916ab7128e4a00c8f16b9ee2c2f9666c198dc8c6749407c2f5d95c1"
Nov 03 19:52:17.368515 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:52:17.368489    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7124099ad916ab7128e4a00c8f16b9ee2c2f9666c198dc8c6749407c2f5d95c1\": container with ID starting with 7124099ad916ab7128e4a00c8f16b9ee2c2f9666c198dc8c6749407c2f5d95c1 not found: ID does not exist" containerID="7124099ad916ab7128e4a00c8f16b9ee2c2f9666c198dc8c6749407c2f5d95c1"
Nov 03 19:52:17.368618 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.368528    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7124099ad916ab7128e4a00c8f16b9ee2c2f9666c198dc8c6749407c2f5d95c1"} err="failed to get container status \"7124099ad916ab7128e4a00c8f16b9ee2c2f9666c198dc8c6749407c2f5d95c1\": rpc error: code = NotFound desc = could not find container \"7124099ad916ab7128e4a00c8f16b9ee2c2f9666c198dc8c6749407c2f5d95c1\": container with ID starting with 7124099ad916ab7128e4a00c8f16b9ee2c2f9666c198dc8c6749407c2f5d95c1 not found: ID does not exist"
Nov 03 19:52:17.438806 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.438760    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"]
Nov 03 19:52:17.442570 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.442535    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:17.442817 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.442792    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b2e60-6847d65649-5gfvg"]
Nov 03 19:52:17.860845 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.860797    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17572c14-846c-4c5d-a141-6d36045da219" path="/var/lib/kubelet/pods/17572c14-846c-4c5d-a141-6d36045da219/volumes"
Nov 03 19:52:17.997380 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:17.997347    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:18.444910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:18.444253    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:18.750287 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:18.750202    5237 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:52:18.750287 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:18.750277    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm" podUID="2b4a25ae-6965-4713-ab67-327317f678d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:52:19.157866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:19.153963    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:19.655901 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:19.655869    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:19.784483 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:19.784448    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:19.927423 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:19.927329    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:20.724654 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:20.724610    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:21.063583 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.063536    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:21.194570 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.194529    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:21.350943 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.350824    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:21.509470 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.506325    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:21.839044 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.839018    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:52:21.886927 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.880537    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:52:21.919422 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.919399    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:52:21.936057 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936026    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-istio-data\") pod \"2b4a25ae-6965-4713-ab67-327317f678d8\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") "
Nov 03 19:52:21.936057 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936060    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-data\") pod \"e63a1c5a-826e-487e-b3eb-eab4278b8049\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") "
Nov 03 19:52:21.936317 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936086    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-workload-socket\") pod \"2b4a25ae-6965-4713-ab67-327317f678d8\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") "
Nov 03 19:52:21.936317 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936114    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-workload-certs\") pod \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") "
Nov 03 19:52:21.936317 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936166    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-workload-socket\") pod \"e63a1c5a-826e-487e-b3eb-eab4278b8049\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") "
Nov 03 19:52:21.936317 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936199    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-token\" (UniqueName: \"kubernetes.io/projected/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-token\") pod \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") "
Nov 03 19:52:21.936317 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936226    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istiod-ca-cert\") pod \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") "
Nov 03 19:52:21.936598 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936276    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-istio-envoy\") pod \"2b4a25ae-6965-4713-ab67-327317f678d8\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") "
Nov 03 19:52:21.936598 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936360    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-data" pod "e63a1c5a-826e-487e-b3eb-eab4278b8049" (UID: "e63a1c5a-826e-487e-b3eb-eab4278b8049"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:21.936598 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936366    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-istio-data" pod "2b4a25ae-6965-4713-ab67-327317f678d8" (UID: "2b4a25ae-6965-4713-ab67-327317f678d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:21.936598 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936397    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-workload-socket\") pod \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") "
Nov 03 19:52:21.936598 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936466    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-credential-socket\") pod \"e63a1c5a-826e-487e-b3eb-eab4278b8049\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") "
Nov 03 19:52:21.936598 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936498    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b4a25ae-6965-4713-ab67-327317f678d8-istio-token\" (UniqueName: \"kubernetes.io/projected/2b4a25ae-6965-4713-ab67-327317f678d8-istio-token\") pod \"2b4a25ae-6965-4713-ab67-327317f678d8\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") "
Nov 03 19:52:21.936598 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936548    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-podinfo\") pod \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") "
Nov 03 19:52:21.936598 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936590    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-workload-socket" pod "e63a1c5a-826e-487e-b3eb-eab4278b8049" (UID: "e63a1c5a-826e-487e-b3eb-eab4278b8049"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:21.937048 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936605    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b4a25ae-6965-4713-ab67-327317f678d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4a25ae-6965-4713-ab67-327317f678d8-istiod-ca-cert\") pod \"2b4a25ae-6965-4713-ab67-327317f678d8\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") "
Nov 03 19:52:21.937048 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936645    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63a1c5a-826e-487e-b3eb-eab4278b8049-kube-api-access-wg8lj\" (UniqueName: \"kubernetes.io/projected/e63a1c5a-826e-487e-b3eb-eab4278b8049-kube-api-access-wg8lj\") pod \"e63a1c5a-826e-487e-b3eb-eab4278b8049\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") "
Nov 03 19:52:21.937048 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936690    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-workload-certs\") pod \"e63a1c5a-826e-487e-b3eb-eab4278b8049\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") "
Nov 03 19:52:21.937048 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936724    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b4a25ae-6965-4713-ab67-327317f678d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4a25ae-6965-4713-ab67-327317f678d8-istio-podinfo\") pod \"2b4a25ae-6965-4713-ab67-327317f678d8\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") "
Nov 03 19:52:21.937048 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936768    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-credential-socket\") pod \"2b4a25ae-6965-4713-ab67-327317f678d8\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") "
Nov 03 19:52:21.937048 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936803    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b4a25ae-6965-4713-ab67-327317f678d8-kube-api-access-z6grv\" (UniqueName: \"kubernetes.io/projected/2b4a25ae-6965-4713-ab67-327317f678d8-kube-api-access-z6grv\") pod \"2b4a25ae-6965-4713-ab67-327317f678d8\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") "
Nov 03 19:52:21.937048 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936807    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istiod-ca-cert" pod "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" (UID: "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:21.937048 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936863    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-data\") pod \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") "
Nov 03 19:52:21.937048 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936905    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-envoy\") pod \"e63a1c5a-826e-487e-b3eb-eab4278b8049\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") "
Nov 03 19:52:21.937048 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936956    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-credential-socket\") pod \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") "
Nov 03 19:52:21.937048 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936996    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-token\" (UniqueName: \"kubernetes.io/projected/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-token\") pod \"e63a1c5a-826e-487e-b3eb-eab4278b8049\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") "
Nov 03 19:52:21.937048 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.936997    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-credential-socket" pod "e63a1c5a-826e-487e-b3eb-eab4278b8049" (UID: "e63a1c5a-826e-487e-b3eb-eab4278b8049"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:21.937048 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.937034    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-podinfo\") pod \"e63a1c5a-826e-487e-b3eb-eab4278b8049\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") "
Nov 03 19:52:21.937712 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.937082    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-envoy\") pod \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") "
Nov 03 19:52:21.937712 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.937139    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-kube-api-access-ft6cn\" (UniqueName: \"kubernetes.io/projected/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-kube-api-access-ft6cn\") pod \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\" (UID: \"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb\") "
Nov 03 19:52:21.937712 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.937225    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-workload-certs\") pod \"2b4a25ae-6965-4713-ab67-327317f678d8\" (UID: \"2b4a25ae-6965-4713-ab67-327317f678d8\") "
Nov 03 19:52:21.937712 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.937268    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63a1c5a-826e-487e-b3eb-eab4278b8049-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63a1c5a-826e-487e-b3eb-eab4278b8049-istiod-ca-cert\") pod \"e63a1c5a-826e-487e-b3eb-eab4278b8049\" (UID: \"e63a1c5a-826e-487e-b3eb-eab4278b8049\") "
Nov 03 19:52:21.937712 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.937281    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-workload-socket" pod "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" (UID: "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:21.938013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.937921    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:21.938013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.937945    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:21.938013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.937960    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:21.938013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.937974    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:21.938013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.937989    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:21.938013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.938005    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:21.938357 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.938197    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-workload-certs" pod "e63a1c5a-826e-487e-b3eb-eab4278b8049" (UID: "e63a1c5a-826e-487e-b3eb-eab4278b8049"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:21.938412 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.938366    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-credential-socket" pod "2b4a25ae-6965-4713-ab67-327317f678d8" (UID: "2b4a25ae-6965-4713-ab67-327317f678d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:21.938521 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.938466    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-workload-certs" pod "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" (UID: "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:21.938613 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.938488    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63a1c5a-826e-487e-b3eb-eab4278b8049-istiod-ca-cert" pod "e63a1c5a-826e-487e-b3eb-eab4278b8049" (UID: "e63a1c5a-826e-487e-b3eb-eab4278b8049"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:21.938613 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.938548    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-istio-envoy" pod "2b4a25ae-6965-4713-ab67-327317f678d8" (UID: "2b4a25ae-6965-4713-ab67-327317f678d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:21.939063 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.939035    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-workload-socket" pod "2b4a25ae-6965-4713-ab67-327317f678d8" (UID: "2b4a25ae-6965-4713-ab67-327317f678d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:21.939186 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.939071    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-credential-socket" pod "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" (UID: "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:21.939186 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.939100    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-workload-certs" pod "2b4a25ae-6965-4713-ab67-327317f678d8" (UID: "2b4a25ae-6965-4713-ab67-327317f678d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:21.939186 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.939141    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-data" pod "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" (UID: "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:21.939648 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.939620    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b4a25ae-6965-4713-ab67-327317f678d8-istiod-ca-cert" pod "2b4a25ae-6965-4713-ab67-327317f678d8" (UID: "2b4a25ae-6965-4713-ab67-327317f678d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:21.941472 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.941447    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-envoy" pod "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" (UID: "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:21.941570 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.941537    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-podinfo" pod "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" (UID: "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:52:21.941816 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.941737    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-kube-api-access-ft6cn" pod "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" (UID: "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb"). InnerVolumeSpecName "kube-api-access-ft6cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:21.941816 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.941773    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63a1c5a-826e-487e-b3eb-eab4278b8049-kube-api-access-wg8lj" pod "e63a1c5a-826e-487e-b3eb-eab4278b8049" (UID: "e63a1c5a-826e-487e-b3eb-eab4278b8049"). InnerVolumeSpecName "kube-api-access-wg8lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:21.941961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.941848    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4a25ae-6965-4713-ab67-327317f678d8-istio-token" pod "2b4a25ae-6965-4713-ab67-327317f678d8" (UID: "2b4a25ae-6965-4713-ab67-327317f678d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:21.941961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.941885    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b4a25ae-6965-4713-ab67-327317f678d8-istio-podinfo" pod "2b4a25ae-6965-4713-ab67-327317f678d8" (UID: "2b4a25ae-6965-4713-ab67-327317f678d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:52:21.942540 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.942521    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-token" pod "e63a1c5a-826e-487e-b3eb-eab4278b8049" (UID: "e63a1c5a-826e-487e-b3eb-eab4278b8049"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:21.943211 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.943190    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-envoy" pod "e63a1c5a-826e-487e-b3eb-eab4278b8049" (UID: "e63a1c5a-826e-487e-b3eb-eab4278b8049"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:21.943401 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.943370    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-token" pod "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" (UID: "3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:21.943401 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.943387    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-podinfo" pod "e63a1c5a-826e-487e-b3eb-eab4278b8049" (UID: "e63a1c5a-826e-487e-b3eb-eab4278b8049"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:52:21.943700 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.943682    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4a25ae-6965-4713-ab67-327317f678d8-kube-api-access-z6grv" pod "2b4a25ae-6965-4713-ab67-327317f678d8" (UID: "2b4a25ae-6965-4713-ab67-327317f678d8"). InnerVolumeSpecName "kube-api-access-z6grv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:21.979293 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:21.979254    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:22.039642 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039606    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039642 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039638    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039642 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039651    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039660    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039668    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b4a25ae-6965-4713-ab67-327317f678d8-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039677    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039688    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4a25ae-6965-4713-ab67-327317f678d8-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039696    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg8lj\" (UniqueName: \"kubernetes.io/projected/e63a1c5a-826e-487e-b3eb-eab4278b8049-kube-api-access-wg8lj\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039706    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039714    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4a25ae-6965-4713-ab67-327317f678d8-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039722    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039730    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6grv\" (UniqueName: \"kubernetes.io/projected/2b4a25ae-6965-4713-ab67-327317f678d8-kube-api-access-z6grv\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039738    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039746    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039755    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039763    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039772    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63a1c5a-826e-487e-b3eb-eab4278b8049-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039780    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039788    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft6cn\" (UniqueName: \"kubernetes.io/projected/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb-kube-api-access-ft6cn\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039796    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4a25ae-6965-4713-ab67-327317f678d8-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.039930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.039804    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63a1c5a-826e-487e-b3eb-eab4278b8049-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:22.184977 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.166775    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:22.399002 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.398962    5237 generic.go:356] "Generic (PLEG): container finished" podID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerID="78334ef3ed948b6a5413211648331cc6760a29bcb35e8e0645bb59022ff64c08" exitCode=0
Nov 03 19:52:22.399159 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.399016    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"
Nov 03 19:52:22.399159 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.399024    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" event={"ID":"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb","Type":"ContainerDied","Data":"78334ef3ed948b6a5413211648331cc6760a29bcb35e8e0645bb59022ff64c08"}
Nov 03 19:52:22.399159 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.399081    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg" event={"ID":"3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb","Type":"ContainerDied","Data":"77bbe6e8626f1cd5c933ab8b61e2d237707bbd04f711fdcaf08680e48f54eaeb"}
Nov 03 19:52:22.399159 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.399102    5237 scope.go:122] "RemoveContainer" containerID="78334ef3ed948b6a5413211648331cc6760a29bcb35e8e0645bb59022ff64c08"
Nov 03 19:52:22.400636 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.400608    5237 generic.go:356] "Generic (PLEG): container finished" podID="e63a1c5a-826e-487e-b3eb-eab4278b8049" containerID="36a30a9ec5c8248309e48633fbf8924b7ec2d1ff29a8eb2e1b39c94a10fe2bc2" exitCode=0
Nov 03 19:52:22.400772 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.400671    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"
Nov 03 19:52:22.400772 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.400690    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" event={"ID":"e63a1c5a-826e-487e-b3eb-eab4278b8049","Type":"ContainerDied","Data":"36a30a9ec5c8248309e48633fbf8924b7ec2d1ff29a8eb2e1b39c94a10fe2bc2"}
Nov 03 19:52:22.400772 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.400761    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k" event={"ID":"e63a1c5a-826e-487e-b3eb-eab4278b8049","Type":"ContainerDied","Data":"fb0069ace7af2b7639c7a27e34996e2bbd9c74543c29bd91881949a5705d6b91"}
Nov 03 19:52:22.402322 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.402300    5237 generic.go:356] "Generic (PLEG): container finished" podID="2b4a25ae-6965-4713-ab67-327317f678d8" containerID="7208ce730ab4cc3e006c968ed14b4def81f8c01b75fb3e5aad77dc714cd1b475" exitCode=0
Nov 03 19:52:22.402444 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.402352    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"
Nov 03 19:52:22.402444 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.402384    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm" event={"ID":"2b4a25ae-6965-4713-ab67-327317f678d8","Type":"ContainerDied","Data":"7208ce730ab4cc3e006c968ed14b4def81f8c01b75fb3e5aad77dc714cd1b475"}
Nov 03 19:52:22.402520 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.402444    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm" event={"ID":"2b4a25ae-6965-4713-ab67-327317f678d8","Type":"ContainerDied","Data":"bd18bb4918ed919fd6fc8272eab5c5fea29ec62a01ad7a832c405ae80ac151d8"}
Nov 03 19:52:22.411991 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.411972    5237 scope.go:122] "RemoveContainer" containerID="78334ef3ed948b6a5413211648331cc6760a29bcb35e8e0645bb59022ff64c08"
Nov 03 19:52:22.412880 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:52:22.412662    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78334ef3ed948b6a5413211648331cc6760a29bcb35e8e0645bb59022ff64c08\": container with ID starting with 78334ef3ed948b6a5413211648331cc6760a29bcb35e8e0645bb59022ff64c08 not found: ID does not exist" containerID="78334ef3ed948b6a5413211648331cc6760a29bcb35e8e0645bb59022ff64c08"
Nov 03 19:52:22.412880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.412693    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78334ef3ed948b6a5413211648331cc6760a29bcb35e8e0645bb59022ff64c08"} err="failed to get container status \"78334ef3ed948b6a5413211648331cc6760a29bcb35e8e0645bb59022ff64c08\": rpc error: code = NotFound desc = could not find container \"78334ef3ed948b6a5413211648331cc6760a29bcb35e8e0645bb59022ff64c08\": container with ID starting with 78334ef3ed948b6a5413211648331cc6760a29bcb35e8e0645bb59022ff64c08 not found: ID does not exist"
Nov 03 19:52:22.412880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.412710    5237 scope.go:122] "RemoveContainer" containerID="36a30a9ec5c8248309e48633fbf8924b7ec2d1ff29a8eb2e1b39c94a10fe2bc2"
Nov 03 19:52:22.431985 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.431960    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"]
Nov 03 19:52:22.440067 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.440043    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-x9f4k"]
Nov 03 19:52:22.445858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.445820    5237 scope.go:122] "RemoveContainer" containerID="36a30a9ec5c8248309e48633fbf8924b7ec2d1ff29a8eb2e1b39c94a10fe2bc2"
Nov 03 19:52:22.446326 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:52:22.446305    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a30a9ec5c8248309e48633fbf8924b7ec2d1ff29a8eb2e1b39c94a10fe2bc2\": container with ID starting with 36a30a9ec5c8248309e48633fbf8924b7ec2d1ff29a8eb2e1b39c94a10fe2bc2 not found: ID does not exist" containerID="36a30a9ec5c8248309e48633fbf8924b7ec2d1ff29a8eb2e1b39c94a10fe2bc2"
Nov 03 19:52:22.446431 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.446335    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a30a9ec5c8248309e48633fbf8924b7ec2d1ff29a8eb2e1b39c94a10fe2bc2"} err="failed to get container status \"36a30a9ec5c8248309e48633fbf8924b7ec2d1ff29a8eb2e1b39c94a10fe2bc2\": rpc error: code = NotFound desc = could not find container \"36a30a9ec5c8248309e48633fbf8924b7ec2d1ff29a8eb2e1b39c94a10fe2bc2\": container with ID starting with 36a30a9ec5c8248309e48633fbf8924b7ec2d1ff29a8eb2e1b39c94a10fe2bc2 not found: ID does not exist"
Nov 03 19:52:22.446431 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.446358    5237 scope.go:122] "RemoveContainer" containerID="7208ce730ab4cc3e006c968ed14b4def81f8c01b75fb3e5aad77dc714cd1b475"
Nov 03 19:52:22.452043 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.452017    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"]
Nov 03 19:52:22.457084 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.457060    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-8mvrg"]
Nov 03 19:52:22.466197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.466173    5237 scope.go:122] "RemoveContainer" containerID="7208ce730ab4cc3e006c968ed14b4def81f8c01b75fb3e5aad77dc714cd1b475"
Nov 03 19:52:22.466690 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:52:22.466666    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7208ce730ab4cc3e006c968ed14b4def81f8c01b75fb3e5aad77dc714cd1b475\": container with ID starting with 7208ce730ab4cc3e006c968ed14b4def81f8c01b75fb3e5aad77dc714cd1b475 not found: ID does not exist" containerID="7208ce730ab4cc3e006c968ed14b4def81f8c01b75fb3e5aad77dc714cd1b475"
Nov 03 19:52:22.466805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.466699    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7208ce730ab4cc3e006c968ed14b4def81f8c01b75fb3e5aad77dc714cd1b475"} err="failed to get container status \"7208ce730ab4cc3e006c968ed14b4def81f8c01b75fb3e5aad77dc714cd1b475\": rpc error: code = NotFound desc = could not find container \"7208ce730ab4cc3e006c968ed14b4def81f8c01b75fb3e5aad77dc714cd1b475\": container with ID starting with 7208ce730ab4cc3e006c968ed14b4def81f8c01b75fb3e5aad77dc714cd1b475 not found: ID does not exist"
Nov 03 19:52:22.472741 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.472717    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"]
Nov 03 19:52:22.477171 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.477150    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-6w6mm"]
Nov 03 19:52:22.606267 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.606233    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"]
Nov 03 19:52:22.619054 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.619017    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"]
Nov 03 19:52:22.720351 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.720273    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:22.854213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:22.854180    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:23.207006 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:23.206501    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:23.854702 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:23.854665    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:23.898886 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:23.894765    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b4a25ae-6965-4713-ab67-327317f678d8" path="/var/lib/kubelet/pods/2b4a25ae-6965-4713-ab67-327317f678d8/volumes"
Nov 03 19:52:23.898886 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:23.895522    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" path="/var/lib/kubelet/pods/3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb/volumes"
Nov 03 19:52:23.898886 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:23.896201    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e63a1c5a-826e-487e-b3eb-eab4278b8049" path="/var/lib/kubelet/pods/e63a1c5a-826e-487e-b3eb-eab4278b8049/volumes"
Nov 03 19:52:24.448607 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:24.448511    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:25.080675 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:25.080635    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:25.220871 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:25.220767    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:25.374577 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:25.374536    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:25.626856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:25.615755    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:25.773934 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:25.773893    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:25.912148 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:25.912064    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:26.219285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:26.219187    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:26.433494 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:26.421553    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:26.791581 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:26.791548    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:26.924578 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:26.924543    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"]
Nov 03 19:52:26.938891 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:26.938863    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"]
Nov 03 19:52:26.958280 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:26.958222    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:27.092880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:27.092747    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:27.607205 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:27.607140    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b" podUID="f6b14de1-629b-4f1a-b35c-cd23b30eff51" containerName="payload-processing" containerID="cri-o://685e181c6cd4f7242881b0c1979e1f07667f513e467c9567660b3be29d4d7d13" gracePeriod=25
Nov 03 19:52:27.619529 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:27.619483    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv" podUID="f4d213de-c3e2-4455-ab48-e78d0f72b169" containerName="payload-pre-processing" containerID="cri-o://54668c64f608b0c67e643a38300b3dde1fed941ee5aa9670d918866418484835" gracePeriod=25
Nov 03 19:52:27.746680 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:27.746610    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:27.923470 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:27.923419    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"
Nov 03 19:52:27.953285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:27.953258    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"
Nov 03 19:52:28.034074 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.034039    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b14de1-629b-4f1a-b35c-cd23b30eff51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b14de1-629b-4f1a-b35c-cd23b30eff51-plugins-config-volume\") pod \"f6b14de1-629b-4f1a-b35c-cd23b30eff51\" (UID: \"f6b14de1-629b-4f1a-b35c-cd23b30eff51\") "
Nov 03 19:52:28.034250 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.034083    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d213de-c3e2-4455-ab48-e78d0f72b169-kube-api-access-tcgzg\" (UniqueName: \"kubernetes.io/projected/f4d213de-c3e2-4455-ab48-e78d0f72b169-kube-api-access-tcgzg\") pod \"f4d213de-c3e2-4455-ab48-e78d0f72b169\" (UID: \"f4d213de-c3e2-4455-ab48-e78d0f72b169\") "
Nov 03 19:52:28.034250 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.034130    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b14de1-629b-4f1a-b35c-cd23b30eff51-kube-api-access-9mc9w\" (UniqueName: \"kubernetes.io/projected/f6b14de1-629b-4f1a-b35c-cd23b30eff51-kube-api-access-9mc9w\") pod \"f6b14de1-629b-4f1a-b35c-cd23b30eff51\" (UID: \"f6b14de1-629b-4f1a-b35c-cd23b30eff51\") "
Nov 03 19:52:28.034250 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.034194    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d213de-c3e2-4455-ab48-e78d0f72b169-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d213de-c3e2-4455-ab48-e78d0f72b169-plugins-config-volume\") pod \"f4d213de-c3e2-4455-ab48-e78d0f72b169\" (UID: \"f4d213de-c3e2-4455-ab48-e78d0f72b169\") "
Nov 03 19:52:28.034547 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.034508    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b14de1-629b-4f1a-b35c-cd23b30eff51-plugins-config-volume" pod "f6b14de1-629b-4f1a-b35c-cd23b30eff51" (UID: "f6b14de1-629b-4f1a-b35c-cd23b30eff51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:28.034686 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.034660    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d213de-c3e2-4455-ab48-e78d0f72b169-plugins-config-volume" pod "f4d213de-c3e2-4455-ab48-e78d0f72b169" (UID: "f4d213de-c3e2-4455-ab48-e78d0f72b169"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:28.034787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.034771    5237 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b14de1-629b-4f1a-b35c-cd23b30eff51-plugins-config-volume\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:28.034852 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.034788    5237 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d213de-c3e2-4455-ab48-e78d0f72b169-plugins-config-volume\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:28.036560 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.036537    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b14de1-629b-4f1a-b35c-cd23b30eff51-kube-api-access-9mc9w" pod "f6b14de1-629b-4f1a-b35c-cd23b30eff51" (UID: "f6b14de1-629b-4f1a-b35c-cd23b30eff51"). InnerVolumeSpecName "kube-api-access-9mc9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:28.036652 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.036589    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d213de-c3e2-4455-ab48-e78d0f72b169-kube-api-access-tcgzg" pod "f4d213de-c3e2-4455-ab48-e78d0f72b169" (UID: "f4d213de-c3e2-4455-ab48-e78d0f72b169"). InnerVolumeSpecName "kube-api-access-tcgzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:28.135649 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.135608    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcgzg\" (UniqueName: \"kubernetes.io/projected/f4d213de-c3e2-4455-ab48-e78d0f72b169-kube-api-access-tcgzg\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:28.135649 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.135648    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mc9w\" (UniqueName: \"kubernetes.io/projected/f6b14de1-629b-4f1a-b35c-cd23b30eff51-kube-api-access-9mc9w\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:28.331218 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.331184    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:28.467830 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.467798    5237 generic.go:356] "Generic (PLEG): container finished" podID="f4d213de-c3e2-4455-ab48-e78d0f72b169" containerID="54668c64f608b0c67e643a38300b3dde1fed941ee5aa9670d918866418484835" exitCode=0
Nov 03 19:52:28.467830 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.467865    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"
Nov 03 19:52:28.468139 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.467871    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv" event={"ID":"f4d213de-c3e2-4455-ab48-e78d0f72b169","Type":"ContainerDied","Data":"54668c64f608b0c67e643a38300b3dde1fed941ee5aa9670d918866418484835"}
Nov 03 19:52:28.468139 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.467937    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv" event={"ID":"f4d213de-c3e2-4455-ab48-e78d0f72b169","Type":"ContainerDied","Data":"0f37bb2d858c6d75e08b1fd5fa9ec8d2374237cea7d8a66250aa1fdfe3f50eef"}
Nov 03 19:52:28.468139 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.467966    5237 scope.go:122] "RemoveContainer" containerID="54668c64f608b0c67e643a38300b3dde1fed941ee5aa9670d918866418484835"
Nov 03 19:52:28.469659 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.469632    5237 generic.go:356] "Generic (PLEG): container finished" podID="f6b14de1-629b-4f1a-b35c-cd23b30eff51" containerID="685e181c6cd4f7242881b0c1979e1f07667f513e467c9567660b3be29d4d7d13" exitCode=0
Nov 03 19:52:28.469820 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.469676    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"
Nov 03 19:52:28.469820 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.469678    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b" event={"ID":"f6b14de1-629b-4f1a-b35c-cd23b30eff51","Type":"ContainerDied","Data":"685e181c6cd4f7242881b0c1979e1f07667f513e467c9567660b3be29d4d7d13"}
Nov 03 19:52:28.469820 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.469785    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b" event={"ID":"f6b14de1-629b-4f1a-b35c-cd23b30eff51","Type":"ContainerDied","Data":"b9ee6b683d9e7a973e703602fe0382254a54fb9c603b48b926397b8a995bdf56"}
Nov 03 19:52:28.479187 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.479168    5237 scope.go:122] "RemoveContainer" containerID="54668c64f608b0c67e643a38300b3dde1fed941ee5aa9670d918866418484835"
Nov 03 19:52:28.479508 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:52:28.479488    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54668c64f608b0c67e643a38300b3dde1fed941ee5aa9670d918866418484835\": container with ID starting with 54668c64f608b0c67e643a38300b3dde1fed941ee5aa9670d918866418484835 not found: ID does not exist" containerID="54668c64f608b0c67e643a38300b3dde1fed941ee5aa9670d918866418484835"
Nov 03 19:52:28.479560 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.479516    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54668c64f608b0c67e643a38300b3dde1fed941ee5aa9670d918866418484835"} err="failed to get container status \"54668c64f608b0c67e643a38300b3dde1fed941ee5aa9670d918866418484835\": rpc error: code = NotFound desc = could not find container \"54668c64f608b0c67e643a38300b3dde1fed941ee5aa9670d918866418484835\": container with ID starting with 54668c64f608b0c67e643a38300b3dde1fed941ee5aa9670d918866418484835 not found: ID does not exist"
Nov 03 19:52:28.479560 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.479535    5237 scope.go:122] "RemoveContainer" containerID="685e181c6cd4f7242881b0c1979e1f07667f513e467c9567660b3be29d4d7d13"
Nov 03 19:52:28.488319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.488300    5237 scope.go:122] "RemoveContainer" containerID="685e181c6cd4f7242881b0c1979e1f07667f513e467c9567660b3be29d4d7d13"
Nov 03 19:52:28.488627 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:52:28.488604    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685e181c6cd4f7242881b0c1979e1f07667f513e467c9567660b3be29d4d7d13\": container with ID starting with 685e181c6cd4f7242881b0c1979e1f07667f513e467c9567660b3be29d4d7d13 not found: ID does not exist" containerID="685e181c6cd4f7242881b0c1979e1f07667f513e467c9567660b3be29d4d7d13"
Nov 03 19:52:28.488706 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.488633    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685e181c6cd4f7242881b0c1979e1f07667f513e467c9567660b3be29d4d7d13"} err="failed to get container status \"685e181c6cd4f7242881b0c1979e1f07667f513e467c9567660b3be29d4d7d13\": rpc error: code = NotFound desc = could not find container \"685e181c6cd4f7242881b0c1979e1f07667f513e467c9567660b3be29d4d7d13\": container with ID starting with 685e181c6cd4f7242881b0c1979e1f07667f513e467c9567660b3be29d4d7d13 not found: ID does not exist"
Nov 03 19:52:28.499311 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.499289    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"]
Nov 03 19:52:28.502266 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.502246    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-25944c-757fcb9b59-bq6dv"]
Nov 03 19:52:28.512684 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.512658    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"]
Nov 03 19:52:28.516664 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:28.516641    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-25944c-7765867fcc-d9k8b"]
Nov 03 19:52:29.042048 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:29.042007    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:29.621801 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:29.621760    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:29.861778 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:29.861741    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d213de-c3e2-4455-ab48-e78d0f72b169" path="/var/lib/kubelet/pods/f4d213de-c3e2-4455-ab48-e78d0f72b169/volumes"
Nov 03 19:52:29.862342 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:29.862318    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b14de1-629b-4f1a-b35c-cd23b30eff51" path="/var/lib/kubelet/pods/f6b14de1-629b-4f1a-b35c-cd23b30eff51/volumes"
Nov 03 19:52:30.326852 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:30.326804    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:30.678145 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:30.678060    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:30.875967 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:30.875929    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:31.011198 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:31.011104    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:31.250877 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:31.250825    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:31.269078 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:31.268135    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"]
Nov 03 19:52:31.291868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:31.286781    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"]
Nov 03 19:52:31.295021 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:31.294988    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"]
Nov 03 19:52:31.412548 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:31.404690    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:31.557293 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:31.557251    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:31.690373 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:31.690331    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:32.318218 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:32.318177    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:32.456994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:32.456957    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:32.803222 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:32.803150    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:32.985465 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:32.985417    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:33.237550 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:33.237456    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:33.616013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:33.615976    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:33.745690 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:33.745655    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:34.553533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:34.553495    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:35.476904 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:35.476776    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:36.115228 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:36.115192    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:36.589891 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:36.589857    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:36.746572 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:36.746536    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:36.901213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:36.901124    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:37.540145 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:37.540104    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:38.317341 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:38.317299    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:38.778498 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:38.778417    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"]
Nov 03 19:52:38.943003 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:38.942969    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:39.105861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:39.096273    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:39.976080 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:39.976044    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:40.609873 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:40.609013    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:40.748515 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:40.748478    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:40.921288 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:40.921204    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:41.094349 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:41.094309    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:41.844320 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:41.844268    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:41.948524 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:41.948481    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main" containerID="cri-o://86a4b06b644d62963636ff81db722847f27161ff9426e6fdea440dbfdb5b3bbc" gracePeriod=45
Nov 03 19:52:41.961524 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:41.961479    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" containerID="cri-o://a38a4a4acdfcc5b146d077911a741036ccc3e5c7fab00970f2545f1c12858df4" gracePeriod=45
Nov 03 19:52:42.237085 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.237054    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:52:42.285528 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.285489    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:52:42.352054 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.352021    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-dshm\") pod \"fae76152-1c87-4685-8074-6bcc1760268c\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") "
Nov 03 19:52:42.353141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.352080    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-home\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-home\") pod \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") "
Nov 03 19:52:42.353141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.352110    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fae76152-1c87-4685-8074-6bcc1760268c-tls-certs\" (UniqueName: \"kubernetes.io/secret/fae76152-1c87-4685-8074-6bcc1760268c-tls-certs\") pod \"fae76152-1c87-4685-8074-6bcc1760268c\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") "
Nov 03 19:52:42.353141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.352145    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-tmp-dir\") pod \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") "
Nov 03 19:52:42.353141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.352212    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-model-cache\") pod \"fae76152-1c87-4685-8074-6bcc1760268c\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") "
Nov 03 19:52:42.353141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.352261    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-tls-certs\") pod \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") "
Nov 03 19:52:42.353141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.352289    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-kube-api-access-66ph5\" (UniqueName: \"kubernetes.io/projected/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-kube-api-access-66ph5\") pod \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") "
Nov 03 19:52:42.353141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.352337    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-home\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-home\") pod \"fae76152-1c87-4685-8074-6bcc1760268c\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") "
Nov 03 19:52:42.353141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.352365    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-dshm\") pod \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") "
Nov 03 19:52:42.353141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.352504    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-tmp-dir\") pod \"fae76152-1c87-4685-8074-6bcc1760268c\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") "
Nov 03 19:52:42.353141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.352545    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-model-cache\") pod \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\" (UID: \"bdf93f15-df9e-4a66-8d89-4d412d3bf35c\") "
Nov 03 19:52:42.353141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.352593    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae76152-1c87-4685-8074-6bcc1760268c-kube-api-access-bg7g7\" (UniqueName: \"kubernetes.io/projected/fae76152-1c87-4685-8074-6bcc1760268c-kube-api-access-bg7g7\") pod \"fae76152-1c87-4685-8074-6bcc1760268c\" (UID: \"fae76152-1c87-4685-8074-6bcc1760268c\") "
Nov 03 19:52:42.353141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.352970    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-model-cache" pod "bdf93f15-df9e-4a66-8d89-4d412d3bf35c" (UID: "bdf93f15-df9e-4a66-8d89-4d412d3bf35c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:42.353141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.353095    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-home" pod "bdf93f15-df9e-4a66-8d89-4d412d3bf35c" (UID: "bdf93f15-df9e-4a66-8d89-4d412d3bf35c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:42.354794 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.353162    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-tmp-dir" pod "bdf93f15-df9e-4a66-8d89-4d412d3bf35c" (UID: "bdf93f15-df9e-4a66-8d89-4d412d3bf35c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:42.354794 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.353328    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-home" pod "fae76152-1c87-4685-8074-6bcc1760268c" (UID: "fae76152-1c87-4685-8074-6bcc1760268c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:42.354794 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.353575    5237 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-home\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:42.354794 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.353587    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-model-cache" pod "fae76152-1c87-4685-8074-6bcc1760268c" (UID: "fae76152-1c87-4685-8074-6bcc1760268c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:42.354794 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.353598    5237 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-tmp-dir\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:42.354794 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.353612    5237 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-home\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:42.354794 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.353627    5237 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-model-cache\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:42.354794 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.353743    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-tmp-dir" pod "fae76152-1c87-4685-8074-6bcc1760268c" (UID: "fae76152-1c87-4685-8074-6bcc1760268c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:42.356133 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.356098    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-dshm" pod "bdf93f15-df9e-4a66-8d89-4d412d3bf35c" (UID: "bdf93f15-df9e-4a66-8d89-4d412d3bf35c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:42.356251 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.356149    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-tls-certs" pod "bdf93f15-df9e-4a66-8d89-4d412d3bf35c" (UID: "bdf93f15-df9e-4a66-8d89-4d412d3bf35c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:42.356251 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.356215    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-kube-api-access-66ph5" pod "bdf93f15-df9e-4a66-8d89-4d412d3bf35c" (UID: "bdf93f15-df9e-4a66-8d89-4d412d3bf35c"). InnerVolumeSpecName "kube-api-access-66ph5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:42.357097 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.357065    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-dshm" pod "fae76152-1c87-4685-8074-6bcc1760268c" (UID: "fae76152-1c87-4685-8074-6bcc1760268c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:42.357651 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.357621    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae76152-1c87-4685-8074-6bcc1760268c-kube-api-access-bg7g7" pod "fae76152-1c87-4685-8074-6bcc1760268c" (UID: "fae76152-1c87-4685-8074-6bcc1760268c"). InnerVolumeSpecName "kube-api-access-bg7g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:42.357746 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.357688    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae76152-1c87-4685-8074-6bcc1760268c-tls-certs" pod "fae76152-1c87-4685-8074-6bcc1760268c" (UID: "fae76152-1c87-4685-8074-6bcc1760268c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:42.455118 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.455032    5237 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-tls-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:42.455118 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.455063    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66ph5\" (UniqueName: \"kubernetes.io/projected/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-kube-api-access-66ph5\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:42.455118 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.455075    5237 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf93f15-df9e-4a66-8d89-4d412d3bf35c-dshm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:42.455118 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.455084    5237 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-tmp-dir\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:42.455118 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.455092    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg7g7\" (UniqueName: \"kubernetes.io/projected/fae76152-1c87-4685-8074-6bcc1760268c-kube-api-access-bg7g7\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:42.455118 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.455100    5237 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-dshm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:42.455118 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.455109    5237 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fae76152-1c87-4685-8074-6bcc1760268c-tls-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:42.455118 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.455117    5237 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae76152-1c87-4685-8074-6bcc1760268c-model-cache\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:42.567933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.567808    5237 generic.go:356] "Generic (PLEG): container finished" podID="fae76152-1c87-4685-8074-6bcc1760268c" containerID="a38a4a4acdfcc5b146d077911a741036ccc3e5c7fab00970f2545f1c12858df4" exitCode=0
Nov 03 19:52:42.568104 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.567891    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"
Nov 03 19:52:42.568771 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.567910    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" event={"ID":"fae76152-1c87-4685-8074-6bcc1760268c","Type":"ContainerDied","Data":"a38a4a4acdfcc5b146d077911a741036ccc3e5c7fab00970f2545f1c12858df4"}
Nov 03 19:52:42.568963 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.568792    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" event={"ID":"fae76152-1c87-4685-8074-6bcc1760268c","Type":"ContainerDied","Data":"f83d00ed835e8b929471c2a520294650bcde2c637bed9eab8a9d46bbc82e5ff0"}
Nov 03 19:52:42.568963 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.568819    5237 scope.go:122] "RemoveContainer" containerID="a38a4a4acdfcc5b146d077911a741036ccc3e5c7fab00970f2545f1c12858df4"
Nov 03 19:52:42.570366 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.570342    5237 generic.go:356] "Generic (PLEG): container finished" podID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerID="86a4b06b644d62963636ff81db722847f27161ff9426e6fdea440dbfdb5b3bbc" exitCode=0
Nov 03 19:52:42.570597 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.570573    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" event={"ID":"bdf93f15-df9e-4a66-8d89-4d412d3bf35c","Type":"ContainerDied","Data":"86a4b06b644d62963636ff81db722847f27161ff9426e6fdea440dbfdb5b3bbc"}
Nov 03 19:52:42.570720 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.570612    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr" event={"ID":"bdf93f15-df9e-4a66-8d89-4d412d3bf35c","Type":"ContainerDied","Data":"4244b6f50ce3840f52556f9e25f4261957078c916172777ca6ea9e441f7fd9ed"}
Nov 03 19:52:42.570720 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.570672    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"
Nov 03 19:52:42.580779 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.580757    5237 scope.go:122] "RemoveContainer" containerID="a38a4a4acdfcc5b146d077911a741036ccc3e5c7fab00970f2545f1c12858df4"
Nov 03 19:52:42.581248 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:52:42.581198    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38a4a4acdfcc5b146d077911a741036ccc3e5c7fab00970f2545f1c12858df4\": container with ID starting with a38a4a4acdfcc5b146d077911a741036ccc3e5c7fab00970f2545f1c12858df4 not found: ID does not exist" containerID="a38a4a4acdfcc5b146d077911a741036ccc3e5c7fab00970f2545f1c12858df4"
Nov 03 19:52:42.581358 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.581257    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38a4a4acdfcc5b146d077911a741036ccc3e5c7fab00970f2545f1c12858df4"} err="failed to get container status \"a38a4a4acdfcc5b146d077911a741036ccc3e5c7fab00970f2545f1c12858df4\": rpc error: code = NotFound desc = could not find container \"a38a4a4acdfcc5b146d077911a741036ccc3e5c7fab00970f2545f1c12858df4\": container with ID starting with a38a4a4acdfcc5b146d077911a741036ccc3e5c7fab00970f2545f1c12858df4 not found: ID does not exist"
Nov 03 19:52:42.581358 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.581278    5237 scope.go:122] "RemoveContainer" containerID="86a4b06b644d62963636ff81db722847f27161ff9426e6fdea440dbfdb5b3bbc"
Nov 03 19:52:42.590729 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.590707    5237 scope.go:122] "RemoveContainer" containerID="86a4b06b644d62963636ff81db722847f27161ff9426e6fdea440dbfdb5b3bbc"
Nov 03 19:52:42.591138 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:52:42.591114    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a4b06b644d62963636ff81db722847f27161ff9426e6fdea440dbfdb5b3bbc\": container with ID starting with 86a4b06b644d62963636ff81db722847f27161ff9426e6fdea440dbfdb5b3bbc not found: ID does not exist" containerID="86a4b06b644d62963636ff81db722847f27161ff9426e6fdea440dbfdb5b3bbc"
Nov 03 19:52:42.591255 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.591148    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a4b06b644d62963636ff81db722847f27161ff9426e6fdea440dbfdb5b3bbc"} err="failed to get container status \"86a4b06b644d62963636ff81db722847f27161ff9426e6fdea440dbfdb5b3bbc\": rpc error: code = NotFound desc = could not find container \"86a4b06b644d62963636ff81db722847f27161ff9426e6fdea440dbfdb5b3bbc\": container with ID starting with 86a4b06b644d62963636ff81db722847f27161ff9426e6fdea440dbfdb5b3bbc not found: ID does not exist"
Nov 03 19:52:42.598308 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.598277    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"]
Nov 03 19:52:42.602011 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.601982    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv"]
Nov 03 19:52:42.615047 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.615018    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"]
Nov 03 19:52:42.624216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.624190    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-05e217/distinct-2-w5-05e217-kserve-5cd788cddc-fl4hr"]
Nov 03 19:52:42.942876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:42.939638    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:43.222319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:43.222226    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-05e217/premium-w5-05e217-kserve-69469cd78f-bqvzv" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:43.257079 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:52:43.257039    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:52:43.626348 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:43.626312    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:43.861083 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:43.861046    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" path="/var/lib/kubelet/pods/bdf93f15-df9e-4a66-8d89-4d412d3bf35c/volumes"
Nov 03 19:52:43.861561 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:43.861546    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae76152-1c87-4685-8074-6bcc1760268c" path="/var/lib/kubelet/pods/fae76152-1c87-4685-8074-6bcc1760268c/volumes"
Nov 03 19:52:44.135414 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:44.135355    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:44.270668 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:44.270635    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:44.415909 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:44.415379    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:45.136480 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:45.136442    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:45.969559 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:45.969513    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:46.366739 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.366700    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" containerID="cri-o://db72d8906736ea92a286407b8bcc6306bfebc46a6ed33f27f479f8e84d659399" gracePeriod=45
Nov 03 19:52:46.389335 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.389297    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" containerID="cri-o://a4ef52f5a43e4fbee6b6008d295bd130841e17851b76879258181a2c20e56666" gracePeriod=45
Nov 03 19:52:46.413769 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.413697    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main" containerID="cri-o://e47ba8c6c8130b1e2eec49ce235ccf6ce9f64990d377839c881891e8ee7f4987" gracePeriod=45
Nov 03 19:52:46.435429 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.435381    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": dial tcp 10.133.0.51:8000: connect: connection refused"
Nov 03 19:52:46.454461 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.454427    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": dial tcp 10.133.0.35:8000: connect: connection refused"
Nov 03 19:52:46.603141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.602954    5237 generic.go:356] "Generic (PLEG): container finished" podID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerID="a4ef52f5a43e4fbee6b6008d295bd130841e17851b76879258181a2c20e56666" exitCode=0
Nov 03 19:52:46.603141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.603071    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" event={"ID":"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2","Type":"ContainerDied","Data":"a4ef52f5a43e4fbee6b6008d295bd130841e17851b76879258181a2c20e56666"}
Nov 03 19:52:46.605287 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.605260    5237 generic.go:356] "Generic (PLEG): container finished" podID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerID="e47ba8c6c8130b1e2eec49ce235ccf6ce9f64990d377839c881891e8ee7f4987" exitCode=0
Nov 03 19:52:46.605414 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.605395    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" event={"ID":"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47","Type":"ContainerDied","Data":"e47ba8c6c8130b1e2eec49ce235ccf6ce9f64990d377839c881891e8ee7f4987"}
Nov 03 19:52:46.607065 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.607035    5237 generic.go:356] "Generic (PLEG): container finished" podID="215c21e1-690b-4a4b-ae98-277e56811956" containerID="db72d8906736ea92a286407b8bcc6306bfebc46a6ed33f27f479f8e84d659399" exitCode=0
Nov 03 19:52:46.607166 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.607086    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" event={"ID":"215c21e1-690b-4a4b-ae98-277e56811956","Type":"ContainerDied","Data":"db72d8906736ea92a286407b8bcc6306bfebc46a6ed33f27f479f8e84d659399"}
Nov 03 19:52:46.777570 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.777533    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:52:46.786058 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.786013    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:52:46.798701 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.798672    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:52:46.812151 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.812114    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:46.946435 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.946343    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-tmp-dir\") pod \"215c21e1-690b-4a4b-ae98-277e56811956\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") "
Nov 03 19:52:46.946435 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.946389    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-dshm\") pod \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") "
Nov 03 19:52:46.946663 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.946463    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-kube-api-access-c2rvl\" (UniqueName: \"kubernetes.io/projected/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-kube-api-access-c2rvl\") pod \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") "
Nov 03 19:52:46.946663 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.946492    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-model-cache\") pod \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") "
Nov 03 19:52:46.946663 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.946519    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-home\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-home\") pod \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") "
Nov 03 19:52:46.946663 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.946544    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-tls-certs\") pod \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") "
Nov 03 19:52:46.946663 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.946568    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/215c21e1-690b-4a4b-ae98-277e56811956-tls-certs\" (UniqueName: \"kubernetes.io/secret/215c21e1-690b-4a4b-ae98-277e56811956-tls-certs\") pod \"215c21e1-690b-4a4b-ae98-277e56811956\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") "
Nov 03 19:52:46.946663 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.946615    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-kube-api-access-lfvj8\" (UniqueName: \"kubernetes.io/projected/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-kube-api-access-lfvj8\") pod \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") "
Nov 03 19:52:46.946663 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.946642    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-home\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-home\") pod \"215c21e1-690b-4a4b-ae98-277e56811956\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") "
Nov 03 19:52:46.947073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.946654    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-tmp-dir" pod "215c21e1-690b-4a4b-ae98-277e56811956" (UID: "215c21e1-690b-4a4b-ae98-277e56811956"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:46.947073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.946691    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-model-cache\") pod \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") "
Nov 03 19:52:46.947073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.946782    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-model-cache" pod "f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" (UID: "f82e4bbb-88bb-4b6b-b75d-d187d68e3c47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:46.947073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.946877    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-tmp-dir\") pod \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") "
Nov 03 19:52:46.947073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.946911    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-dshm\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-dshm\") pod \"215c21e1-690b-4a4b-ae98-277e56811956\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") "
Nov 03 19:52:46.947073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.946949    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-tls-certs\" (UniqueName: \"kubernetes.io/secret/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-tls-certs\") pod \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\" (UID: \"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47\") "
Nov 03 19:52:46.947073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.947003    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-tmp-dir\") pod \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") "
Nov 03 19:52:46.947073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.947027    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-model-cache" pod "51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" (UID: "51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:46.947073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.947045    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-home\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-home\") pod \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") "
Nov 03 19:52:46.947502 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.947079    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/215c21e1-690b-4a4b-ae98-277e56811956-kube-api-access-2k2bh\" (UniqueName: \"kubernetes.io/projected/215c21e1-690b-4a4b-ae98-277e56811956-kube-api-access-2k2bh\") pod \"215c21e1-690b-4a4b-ae98-277e56811956\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") "
Nov 03 19:52:46.947502 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.947180    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-tmp-dir" pod "f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" (UID: "f82e4bbb-88bb-4b6b-b75d-d187d68e3c47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:46.947502 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.947205    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-model-cache\") pod \"215c21e1-690b-4a4b-ae98-277e56811956\" (UID: \"215c21e1-690b-4a4b-ae98-277e56811956\") "
Nov 03 19:52:46.947502 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.947258    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-dshm\") pod \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\" (UID: \"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2\") "
Nov 03 19:52:46.947653 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.947373    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-home" pod "f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" (UID: "f82e4bbb-88bb-4b6b-b75d-d187d68e3c47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:46.948155 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.948126    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-tmp-dir" pod "51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" (UID: "51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:46.948885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.948533    5237 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-model-cache\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:46.948885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.948559    5237 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-home\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:46.948885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.948575    5237 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-model-cache\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:46.948885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.948592    5237 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-tmp-dir\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:46.948885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.948607    5237 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-tmp-dir\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:46.948885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.948618    5237 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-tmp-dir\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:46.949418 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.949388    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/215c21e1-690b-4a4b-ae98-277e56811956-tls-certs" pod "215c21e1-690b-4a4b-ae98-277e56811956" (UID: "215c21e1-690b-4a4b-ae98-277e56811956"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:46.949418 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.949407    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-dshm" pod "f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" (UID: "f82e4bbb-88bb-4b6b-b75d-d187d68e3c47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:46.949605 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.949421    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-kube-api-access-c2rvl" pod "f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" (UID: "f82e4bbb-88bb-4b6b-b75d-d187d68e3c47"). InnerVolumeSpecName "kube-api-access-c2rvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:46.949695 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.949673    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-model-cache" pod "215c21e1-690b-4a4b-ae98-277e56811956" (UID: "215c21e1-690b-4a4b-ae98-277e56811956"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:46.949761 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.949699    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-kube-api-access-lfvj8" pod "51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" (UID: "51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2"). InnerVolumeSpecName "kube-api-access-lfvj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:46.949761 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.949723    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-home" pod "51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" (UID: "51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:46.949914 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.949818    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-dshm" pod "51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" (UID: "51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:46.949978 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.949918    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-home" pod "215c21e1-690b-4a4b-ae98-277e56811956" (UID: "215c21e1-690b-4a4b-ae98-277e56811956"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:46.950126 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.950104    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-tls-certs" pod "51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" (UID: "51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:46.950405 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.950389    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-dshm" pod "215c21e1-690b-4a4b-ae98-277e56811956" (UID: "215c21e1-690b-4a4b-ae98-277e56811956"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:46.951303 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.951280    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-tls-certs" pod "f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" (UID: "f82e4bbb-88bb-4b6b-b75d-d187d68e3c47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:46.951398 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:46.951382    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215c21e1-690b-4a4b-ae98-277e56811956-kube-api-access-2k2bh" pod "215c21e1-690b-4a4b-ae98-277e56811956" (UID: "215c21e1-690b-4a4b-ae98-277e56811956"). InnerVolumeSpecName "kube-api-access-2k2bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:47.049488 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.049455    5237 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-dshm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:47.049488 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.049485    5237 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-tls-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:47.049488 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.049494    5237 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-home\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:47.049971 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.049504    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k2bh\" (UniqueName: \"kubernetes.io/projected/215c21e1-690b-4a4b-ae98-277e56811956-kube-api-access-2k2bh\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:47.049971 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.049513    5237 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-model-cache\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:47.049971 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.049522    5237 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-dshm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:47.049971 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.049530    5237 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-dshm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:47.049971 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.049539    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2rvl\" (UniqueName: \"kubernetes.io/projected/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47-kube-api-access-c2rvl\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:47.049971 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.049548    5237 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-tls-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:47.049971 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.049557    5237 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215c21e1-690b-4a4b-ae98-277e56811956-tls-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:47.049971 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.049565    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfvj8\" (UniqueName: \"kubernetes.io/projected/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2-kube-api-access-lfvj8\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:47.049971 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.049573    5237 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215c21e1-690b-4a4b-ae98-277e56811956-home\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:47.549883 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.538699    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:47.614807 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.614776    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"
Nov 03 19:52:47.614807 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.614801    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm" event={"ID":"215c21e1-690b-4a4b-ae98-277e56811956","Type":"ContainerDied","Data":"f6b4c9dcc15e5abf60441ce0009181f65b0f6e8e899b7e03feed4e367b23da38"}
Nov 03 19:52:47.615162 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.614930    5237 scope.go:122] "RemoveContainer" containerID="db72d8906736ea92a286407b8bcc6306bfebc46a6ed33f27f479f8e84d659399"
Nov 03 19:52:47.616356 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.616337    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"
Nov 03 19:52:47.616457 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.616378    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf" event={"ID":"51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2","Type":"ContainerDied","Data":"68d3d5e5908de9bd274bd1ea47bc272e336585284717ba954d3220c92418c80f"}
Nov 03 19:52:47.617764 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.617744    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd" event={"ID":"f82e4bbb-88bb-4b6b-b75d-d187d68e3c47","Type":"ContainerDied","Data":"e604af74767db8885df017d9616424bf30d408f254c91643d1c9c198646c8e3d"}
Nov 03 19:52:47.617764 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.617754    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"
Nov 03 19:52:47.626569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.626475    5237 scope.go:122] "RemoveContainer" containerID="a4ef52f5a43e4fbee6b6008d295bd130841e17851b76879258181a2c20e56666"
Nov 03 19:52:47.636594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.636577    5237 scope.go:122] "RemoveContainer" containerID="e47ba8c6c8130b1e2eec49ce235ccf6ce9f64990d377839c881891e8ee7f4987"
Nov 03 19:52:47.645741 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.645712    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"]
Nov 03 19:52:47.652096 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.652070    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-036184/distinct-2-w3-036184-kserve-5fc865fdbf-fkwbm"]
Nov 03 19:52:47.663981 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.663956    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"]
Nov 03 19:52:47.670992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.670965    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-036184/premium-w3-036184-kserve-8d6bff5d7-mzhxd"]
Nov 03 19:52:47.682273 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.682246    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"]
Nov 03 19:52:47.686674 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.686646    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-036184/simulator-w3-036184-kserve-7bb7df566-txzvf"]
Nov 03 19:52:47.861076 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.861039    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="215c21e1-690b-4a4b-ae98-277e56811956" path="/var/lib/kubelet/pods/215c21e1-690b-4a4b-ae98-277e56811956/volumes"
Nov 03 19:52:47.861528 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.861509    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" path="/var/lib/kubelet/pods/51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2/volumes"
Nov 03 19:52:47.862120 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:47.862098    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" path="/var/lib/kubelet/pods/f82e4bbb-88bb-4b6b-b75d-d187d68e3c47/volumes"
Nov 03 19:52:48.260857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:48.259993    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:49.055451 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:49.055403    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:49.903801 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:49.903749    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:50.875338 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:50.875300    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:51.612556 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:51.612525    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"]
Nov 03 19:52:51.692369 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:51.692332    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:51.970542 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:51.970459    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:52.539094 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:52.539058    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"]
Nov 03 19:52:52.539576 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:52.539540    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7" podUID="93bee0fc-7374-4419-ac8a-07745bdc3a24" containerName="maas-api" containerID="cri-o://2049428a0e150b3e75e258dd7bf2abf7fabd6795db7445d016203ee05185aff3" gracePeriod=30
Nov 03 19:52:52.619998 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:52.619961    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:52.665360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:52.665326    5237 generic.go:356] "Generic (PLEG): container finished" podID="93bee0fc-7374-4419-ac8a-07745bdc3a24" containerID="2049428a0e150b3e75e258dd7bf2abf7fabd6795db7445d016203ee05185aff3" exitCode=0
Nov 03 19:52:52.665522 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:52.665401    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7" event={"ID":"93bee0fc-7374-4419-ac8a-07745bdc3a24","Type":"ContainerDied","Data":"2049428a0e150b3e75e258dd7bf2abf7fabd6795db7445d016203ee05185aff3"}
Nov 03 19:52:52.805793 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:52.805760    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:52.828487 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:52.823268    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:52:52.971861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:52.971385    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93bee0fc-7374-4419-ac8a-07745bdc3a24-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93bee0fc-7374-4419-ac8a-07745bdc3a24-metrics-certs\") pod \"93bee0fc-7374-4419-ac8a-07745bdc3a24\" (UID: \"93bee0fc-7374-4419-ac8a-07745bdc3a24\") "
Nov 03 19:52:52.971861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:52.971535    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93bee0fc-7374-4419-ac8a-07745bdc3a24-kube-api-access-45zgg\" (UniqueName: \"kubernetes.io/projected/93bee0fc-7374-4419-ac8a-07745bdc3a24-kube-api-access-45zgg\") pod \"93bee0fc-7374-4419-ac8a-07745bdc3a24\" (UID: \"93bee0fc-7374-4419-ac8a-07745bdc3a24\") "
Nov 03 19:52:52.971861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:52.971587    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93bee0fc-7374-4419-ac8a-07745bdc3a24-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93bee0fc-7374-4419-ac8a-07745bdc3a24-maas-api-tls\") pod \"93bee0fc-7374-4419-ac8a-07745bdc3a24\" (UID: \"93bee0fc-7374-4419-ac8a-07745bdc3a24\") "
Nov 03 19:52:52.971861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:52.971626    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93bee0fc-7374-4419-ac8a-07745bdc3a24-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bee0fc-7374-4419-ac8a-07745bdc3a24-openshift-service-ca\") pod \"93bee0fc-7374-4419-ac8a-07745bdc3a24\" (UID: \"93bee0fc-7374-4419-ac8a-07745bdc3a24\") "
Nov 03 19:52:53.010896 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:52.996080    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93bee0fc-7374-4419-ac8a-07745bdc3a24-openshift-service-ca" pod "93bee0fc-7374-4419-ac8a-07745bdc3a24" (UID: "93bee0fc-7374-4419-ac8a-07745bdc3a24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:53.010896 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:52.996593    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:53.010896 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.001675    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93bee0fc-7374-4419-ac8a-07745bdc3a24-metrics-certs" pod "93bee0fc-7374-4419-ac8a-07745bdc3a24" (UID: "93bee0fc-7374-4419-ac8a-07745bdc3a24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:53.010896 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.007504    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bee0fc-7374-4419-ac8a-07745bdc3a24-kube-api-access-45zgg" pod "93bee0fc-7374-4419-ac8a-07745bdc3a24" (UID: "93bee0fc-7374-4419-ac8a-07745bdc3a24"). InnerVolumeSpecName "kube-api-access-45zgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:53.025867 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.021306    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93bee0fc-7374-4419-ac8a-07745bdc3a24-maas-api-tls" pod "93bee0fc-7374-4419-ac8a-07745bdc3a24" (UID: "93bee0fc-7374-4419-ac8a-07745bdc3a24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:53.076180 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.075944    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93bee0fc-7374-4419-ac8a-07745bdc3a24-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:53.076180 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.075976    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45zgg\" (UniqueName: \"kubernetes.io/projected/93bee0fc-7374-4419-ac8a-07745bdc3a24-kube-api-access-45zgg\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:53.076180 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.075997    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93bee0fc-7374-4419-ac8a-07745bdc3a24-maas-api-tls\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:53.076180 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.076043    5237 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bee0fc-7374-4419-ac8a-07745bdc3a24-openshift-service-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:53.441681 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.441596    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:53.673257 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.673224    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"
Nov 03 19:52:53.673733 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.673255    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7" event={"ID":"93bee0fc-7374-4419-ac8a-07745bdc3a24","Type":"ContainerDied","Data":"59fe8b164f1f78fc68ba3a719badf5f2696c50b94a2b55730cf4fa7f568a2f96"}
Nov 03 19:52:53.673733 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.673329    5237 scope.go:122] "RemoveContainer" containerID="2049428a0e150b3e75e258dd7bf2abf7fabd6795db7445d016203ee05185aff3"
Nov 03 19:52:53.703434 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.703363    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"]
Nov 03 19:52:53.707578 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.707554    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-763a69-5f7856ddc6-8tfs7"]
Nov 03 19:52:53.801436 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.801363    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main" containerID="cri-o://2370a74e20e21ca8220c8aba110131865df064d692900e077b953658c863d434" gracePeriod=45
Nov 03 19:52:53.864173 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.864132    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93bee0fc-7374-4419-ac8a-07745bdc3a24" path="/var/lib/kubelet/pods/93bee0fc-7374-4419-ac8a-07745bdc3a24/volumes"
Nov 03 19:52:53.937760 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.937725    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:53.973601 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.969928    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"]
Nov 03 19:52:53.973601 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:53.970167    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b" podUID="f48a7944-1a16-42d6-8483-e6d2533ca2b1" containerName="maas-api" containerID="cri-o://a6e0a12cf8d1de9d6c5156f87dd314a10ae094c7f3cda6cbb684fd66824df675" gracePeriod=30
Nov 03 19:52:54.111580 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.111462    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:54.165126 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.163449    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:52:54.247276 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.247179    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:54.293284 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.293261    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb974992-d3ab-4992-8ce4-d639d550163e-kube-api-access-6h7sl\" (UniqueName: \"kubernetes.io/projected/cb974992-d3ab-4992-8ce4-d639d550163e-kube-api-access-6h7sl\") pod \"cb974992-d3ab-4992-8ce4-d639d550163e\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") "
Nov 03 19:52:54.293429 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.293325    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb974992-d3ab-4992-8ce4-d639d550163e-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb974992-d3ab-4992-8ce4-d639d550163e-tls-certs\") pod \"cb974992-d3ab-4992-8ce4-d639d550163e\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") "
Nov 03 19:52:54.293490 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.293447    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-dshm\") pod \"cb974992-d3ab-4992-8ce4-d639d550163e\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") "
Nov 03 19:52:54.293490 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.293479    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-home\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-home\") pod \"cb974992-d3ab-4992-8ce4-d639d550163e\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") "
Nov 03 19:52:54.293602 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.293581    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-model-cache\") pod \"cb974992-d3ab-4992-8ce4-d639d550163e\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") "
Nov 03 19:52:54.293663 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.293633    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-tmp-dir\") pod \"cb974992-d3ab-4992-8ce4-d639d550163e\" (UID: \"cb974992-d3ab-4992-8ce4-d639d550163e\") "
Nov 03 19:52:54.295450 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.295419    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-tmp-dir" pod "cb974992-d3ab-4992-8ce4-d639d550163e" (UID: "cb974992-d3ab-4992-8ce4-d639d550163e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:54.295793 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.295762    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-home" pod "cb974992-d3ab-4992-8ce4-d639d550163e" (UID: "cb974992-d3ab-4992-8ce4-d639d550163e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:54.295952 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.295928    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-model-cache" pod "cb974992-d3ab-4992-8ce4-d639d550163e" (UID: "cb974992-d3ab-4992-8ce4-d639d550163e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:54.296326 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.296304    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:52:54.297443 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.297418    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb974992-d3ab-4992-8ce4-d639d550163e-tls-certs" pod "cb974992-d3ab-4992-8ce4-d639d550163e" (UID: "cb974992-d3ab-4992-8ce4-d639d550163e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:54.298396 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.298377    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-dshm" pod "cb974992-d3ab-4992-8ce4-d639d550163e" (UID: "cb974992-d3ab-4992-8ce4-d639d550163e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:54.299566 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.299536    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb974992-d3ab-4992-8ce4-d639d550163e-kube-api-access-6h7sl" pod "cb974992-d3ab-4992-8ce4-d639d550163e" (UID: "cb974992-d3ab-4992-8ce4-d639d550163e"). InnerVolumeSpecName "kube-api-access-6h7sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:54.396465 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.396428    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f48a7944-1a16-42d6-8483-e6d2533ca2b1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f48a7944-1a16-42d6-8483-e6d2533ca2b1-maas-api-tls\") pod \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\" (UID: \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\") "
Nov 03 19:52:54.396693 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.396667    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f48a7944-1a16-42d6-8483-e6d2533ca2b1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f48a7944-1a16-42d6-8483-e6d2533ca2b1-metrics-certs\") pod \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\" (UID: \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\") "
Nov 03 19:52:54.396782 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.396710    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f48a7944-1a16-42d6-8483-e6d2533ca2b1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f48a7944-1a16-42d6-8483-e6d2533ca2b1-openshift-service-ca\") pod \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\" (UID: \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\") "
Nov 03 19:52:54.396782 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.396768    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f48a7944-1a16-42d6-8483-e6d2533ca2b1-kube-api-access-44ncq\" (UniqueName: \"kubernetes.io/projected/f48a7944-1a16-42d6-8483-e6d2533ca2b1-kube-api-access-44ncq\") pod \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\" (UID: \"f48a7944-1a16-42d6-8483-e6d2533ca2b1\") "
Nov 03 19:52:54.397323 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.397294    5237 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-dshm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:54.397443 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.397328    5237 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-home\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:54.397443 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.397343    5237 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-model-cache\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:54.397443 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.397356    5237 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb974992-d3ab-4992-8ce4-d639d550163e-tmp-dir\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:54.397443 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.397371    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h7sl\" (UniqueName: \"kubernetes.io/projected/cb974992-d3ab-4992-8ce4-d639d550163e-kube-api-access-6h7sl\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:54.397443 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.397385    5237 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb974992-d3ab-4992-8ce4-d639d550163e-tls-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:54.397732 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.397699    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f48a7944-1a16-42d6-8483-e6d2533ca2b1-openshift-service-ca" pod "f48a7944-1a16-42d6-8483-e6d2533ca2b1" (UID: "f48a7944-1a16-42d6-8483-e6d2533ca2b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:54.399053 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.399020    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f48a7944-1a16-42d6-8483-e6d2533ca2b1-maas-api-tls" pod "f48a7944-1a16-42d6-8483-e6d2533ca2b1" (UID: "f48a7944-1a16-42d6-8483-e6d2533ca2b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:54.399171 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.399060    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f48a7944-1a16-42d6-8483-e6d2533ca2b1-metrics-certs" pod "f48a7944-1a16-42d6-8483-e6d2533ca2b1" (UID: "f48a7944-1a16-42d6-8483-e6d2533ca2b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:54.399292 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.399267    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48a7944-1a16-42d6-8483-e6d2533ca2b1-kube-api-access-44ncq" pod "f48a7944-1a16-42d6-8483-e6d2533ca2b1" (UID: "f48a7944-1a16-42d6-8483-e6d2533ca2b1"). InnerVolumeSpecName "kube-api-access-44ncq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:54.401341 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.401315    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:54.499475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.499386    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f48a7944-1a16-42d6-8483-e6d2533ca2b1-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:54.499475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.499423    5237 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f48a7944-1a16-42d6-8483-e6d2533ca2b1-openshift-service-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:54.499475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.499442    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44ncq\" (UniqueName: \"kubernetes.io/projected/f48a7944-1a16-42d6-8483-e6d2533ca2b1-kube-api-access-44ncq\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:54.499475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.499457    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f48a7944-1a16-42d6-8483-e6d2533ca2b1-maas-api-tls\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:54.689898 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.688132    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:54.702869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.701189    5237 generic.go:356] "Generic (PLEG): container finished" podID="f48a7944-1a16-42d6-8483-e6d2533ca2b1" containerID="a6e0a12cf8d1de9d6c5156f87dd314a10ae094c7f3cda6cbb684fd66824df675" exitCode=0
Nov 03 19:52:54.702869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.701384    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b" event={"ID":"f48a7944-1a16-42d6-8483-e6d2533ca2b1","Type":"ContainerDied","Data":"a6e0a12cf8d1de9d6c5156f87dd314a10ae094c7f3cda6cbb684fd66824df675"}
Nov 03 19:52:54.702869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.701432    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b" event={"ID":"f48a7944-1a16-42d6-8483-e6d2533ca2b1","Type":"ContainerDied","Data":"27a90eb63e37b1dd713655b8e9c04a117f0c1afb484d8bcf2a9e51e3e8ea1a8f"}
Nov 03 19:52:54.702869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.701461    5237 scope.go:122] "RemoveContainer" containerID="a6e0a12cf8d1de9d6c5156f87dd314a10ae094c7f3cda6cbb684fd66824df675"
Nov 03 19:52:54.702869 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.701554    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"
Nov 03 19:52:54.721283 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.721245    5237 generic.go:356] "Generic (PLEG): container finished" podID="cb974992-d3ab-4992-8ce4-d639d550163e" containerID="2370a74e20e21ca8220c8aba110131865df064d692900e077b953658c863d434" exitCode=0
Nov 03 19:52:54.721484 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.721437    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" event={"ID":"cb974992-d3ab-4992-8ce4-d639d550163e","Type":"ContainerDied","Data":"2370a74e20e21ca8220c8aba110131865df064d692900e077b953658c863d434"}
Nov 03 19:52:54.721569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.721511    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b" event={"ID":"cb974992-d3ab-4992-8ce4-d639d550163e","Type":"ContainerDied","Data":"2ef4f8453a01d51a7bf98f9fcef632e9ae7d507aa6c029ef303c45e9b10e0b5c"}
Nov 03 19:52:54.721625 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.721581    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"
Nov 03 19:52:54.732786 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.732759    5237 scope.go:122] "RemoveContainer" containerID="a6e0a12cf8d1de9d6c5156f87dd314a10ae094c7f3cda6cbb684fd66824df675"
Nov 03 19:52:54.733460 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:52:54.733435    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e0a12cf8d1de9d6c5156f87dd314a10ae094c7f3cda6cbb684fd66824df675\": container with ID starting with a6e0a12cf8d1de9d6c5156f87dd314a10ae094c7f3cda6cbb684fd66824df675 not found: ID does not exist" containerID="a6e0a12cf8d1de9d6c5156f87dd314a10ae094c7f3cda6cbb684fd66824df675"
Nov 03 19:52:54.733601 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.733578    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e0a12cf8d1de9d6c5156f87dd314a10ae094c7f3cda6cbb684fd66824df675"} err="failed to get container status \"a6e0a12cf8d1de9d6c5156f87dd314a10ae094c7f3cda6cbb684fd66824df675\": rpc error: code = NotFound desc = could not find container \"a6e0a12cf8d1de9d6c5156f87dd314a10ae094c7f3cda6cbb684fd66824df675\": container with ID starting with a6e0a12cf8d1de9d6c5156f87dd314a10ae094c7f3cda6cbb684fd66824df675 not found: ID does not exist"
Nov 03 19:52:54.733688 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.733677    5237 scope.go:122] "RemoveContainer" containerID="2370a74e20e21ca8220c8aba110131865df064d692900e077b953658c863d434"
Nov 03 19:52:54.775924 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.775891    5237 scope.go:122] "RemoveContainer" containerID="2370a74e20e21ca8220c8aba110131865df064d692900e077b953658c863d434"
Nov 03 19:52:54.776774 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:52:54.776736    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2370a74e20e21ca8220c8aba110131865df064d692900e077b953658c863d434\": container with ID starting with 2370a74e20e21ca8220c8aba110131865df064d692900e077b953658c863d434 not found: ID does not exist" containerID="2370a74e20e21ca8220c8aba110131865df064d692900e077b953658c863d434"
Nov 03 19:52:54.782721 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.782698    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"]
Nov 03 19:52:54.783004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.782775    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2370a74e20e21ca8220c8aba110131865df064d692900e077b953658c863d434"} err="failed to get container status \"2370a74e20e21ca8220c8aba110131865df064d692900e077b953658c863d434\": rpc error: code = NotFound desc = could not find container \"2370a74e20e21ca8220c8aba110131865df064d692900e077b953658c863d434\": container with ID starting with 2370a74e20e21ca8220c8aba110131865df064d692900e077b953658c863d434 not found: ID does not exist"
Nov 03 19:52:54.791828 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.791787    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-036184-645546498b-ml69b"]
Nov 03 19:52:54.807942 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.807891    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"]
Nov 03 19:52:54.817180 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.817134    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/simulator-w2-435bd8-kserve-56c69bcd9f-zb55b"]
Nov 03 19:52:54.825570 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:54.825540    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:55.108425 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:55.108390    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:55.640249 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:55.640210    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:55.859264 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:55.859233    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" path="/var/lib/kubelet/pods/cb974992-d3ab-4992-8ce4-d639d550163e/volumes"
Nov 03 19:52:55.859757 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:55.859740    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f48a7944-1a16-42d6-8483-e6d2533ca2b1" path="/var/lib/kubelet/pods/f48a7944-1a16-42d6-8483-e6d2533ca2b1/volumes"
Nov 03 19:52:56.325612 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:56.325574    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:56.612918 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:56.612807    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227" podUID="3882ca5a-6385-41e0-892c-21f22d58ddc9" containerName="payload-processing" containerID="cri-o://cfd97b8b32ad10cb8935759925e24ac4782719a685c08cfd998e432968cbb223" gracePeriod=25
Nov 03 19:52:56.736249 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:56.736224    5237 generic.go:356] "Generic (PLEG): container finished" podID="3882ca5a-6385-41e0-892c-21f22d58ddc9" containerID="cfd97b8b32ad10cb8935759925e24ac4782719a685c08cfd998e432968cbb223" exitCode=0
Nov 03 19:52:56.736356 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:56.736294    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227" event={"ID":"3882ca5a-6385-41e0-892c-21f22d58ddc9","Type":"ContainerDied","Data":"cfd97b8b32ad10cb8935759925e24ac4782719a685c08cfd998e432968cbb223"}
Nov 03 19:52:56.752704 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:56.752664    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"]
Nov 03 19:52:56.762889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:56.752940    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68" podUID="8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae" containerName="maas-api" containerID="cri-o://825a6dd97ad1b2a6cb1b31a1267c4636ac4a7bba986165f4474570890b06976b" gracePeriod=30
Nov 03 19:52:56.842074 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:56.842033    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:56.972872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:56.972082    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"
Nov 03 19:52:57.053866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.048977    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3882ca5a-6385-41e0-892c-21f22d58ddc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3882ca5a-6385-41e0-892c-21f22d58ddc9-plugins-config-volume\") pod \"3882ca5a-6385-41e0-892c-21f22d58ddc9\" (UID: \"3882ca5a-6385-41e0-892c-21f22d58ddc9\") "
Nov 03 19:52:57.053866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.049198    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3882ca5a-6385-41e0-892c-21f22d58ddc9-kube-api-access-2gbb4\" (UniqueName: \"kubernetes.io/projected/3882ca5a-6385-41e0-892c-21f22d58ddc9-kube-api-access-2gbb4\") pod \"3882ca5a-6385-41e0-892c-21f22d58ddc9\" (UID: \"3882ca5a-6385-41e0-892c-21f22d58ddc9\") "
Nov 03 19:52:57.053866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.051340    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:57.053866 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.051602    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3882ca5a-6385-41e0-892c-21f22d58ddc9-plugins-config-volume" pod "3882ca5a-6385-41e0-892c-21f22d58ddc9" (UID: "3882ca5a-6385-41e0-892c-21f22d58ddc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:57.103981 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.093939    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3882ca5a-6385-41e0-892c-21f22d58ddc9-kube-api-access-2gbb4" pod "3882ca5a-6385-41e0-892c-21f22d58ddc9" (UID: "3882ca5a-6385-41e0-892c-21f22d58ddc9"). InnerVolumeSpecName "kube-api-access-2gbb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:57.155917 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.152071    5237 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3882ca5a-6385-41e0-892c-21f22d58ddc9-plugins-config-volume\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:57.155917 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.152115    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gbb4\" (UniqueName: \"kubernetes.io/projected/3882ca5a-6385-41e0-892c-21f22d58ddc9-kube-api-access-2gbb4\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:57.238372 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.238327    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:52:57.252904 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.252871    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-maas-api-tls\") pod \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\" (UID: \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\") "
Nov 03 19:52:57.253091 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.252925    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-openshift-service-ca\") pod \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\" (UID: \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\") "
Nov 03 19:52:57.253091 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.252964    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-metrics-certs\") pod \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\" (UID: \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\") "
Nov 03 19:52:57.253225 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.253179    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-kube-api-access-29vs7\" (UniqueName: \"kubernetes.io/projected/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-kube-api-access-29vs7\") pod \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\" (UID: \"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae\") "
Nov 03 19:52:57.253338 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.253308    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-openshift-service-ca" pod "8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae" (UID: "8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:57.253585 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.253560    5237 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-openshift-service-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:57.255052 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.255029    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-maas-api-tls" pod "8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae" (UID: "8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:57.255141 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.255058    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-metrics-certs" pod "8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae" (UID: "8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:57.255180 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.255165    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-kube-api-access-29vs7" pod "8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae" (UID: "8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae"). InnerVolumeSpecName "kube-api-access-29vs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:57.278370 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.278319    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:57.355246 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.355206    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29vs7\" (UniqueName: \"kubernetes.io/projected/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-kube-api-access-29vs7\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:57.355246 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.355237    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-maas-api-tls\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:57.355246 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.355247    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:57.429566 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.429468    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:57.743646 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.743562    5237 generic.go:356] "Generic (PLEG): container finished" podID="8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae" containerID="825a6dd97ad1b2a6cb1b31a1267c4636ac4a7bba986165f4474570890b06976b" exitCode=0
Nov 03 19:52:57.743646 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.743614    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"
Nov 03 19:52:57.743943 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.743644    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68" event={"ID":"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae","Type":"ContainerDied","Data":"825a6dd97ad1b2a6cb1b31a1267c4636ac4a7bba986165f4474570890b06976b"}
Nov 03 19:52:57.743943 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.743693    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68" event={"ID":"8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae","Type":"ContainerDied","Data":"2324e79e0ffbf7167a182df3c9d58595d9b1a93ccc9f4385deab930f141d0630"}
Nov 03 19:52:57.743943 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.743721    5237 scope.go:122] "RemoveContainer" containerID="825a6dd97ad1b2a6cb1b31a1267c4636ac4a7bba986165f4474570890b06976b"
Nov 03 19:52:57.745285 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.745266    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227" event={"ID":"3882ca5a-6385-41e0-892c-21f22d58ddc9","Type":"ContainerDied","Data":"932ff957e35f5cb79c13414482493bc41ee0e60f487dd105d5be2f42bb8809b7"}
Nov 03 19:52:57.745376 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.745290    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"
Nov 03 19:52:57.758057 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.757939    5237 scope.go:122] "RemoveContainer" containerID="825a6dd97ad1b2a6cb1b31a1267c4636ac4a7bba986165f4474570890b06976b"
Nov 03 19:52:57.758638 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:52:57.758607    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825a6dd97ad1b2a6cb1b31a1267c4636ac4a7bba986165f4474570890b06976b\": container with ID starting with 825a6dd97ad1b2a6cb1b31a1267c4636ac4a7bba986165f4474570890b06976b not found: ID does not exist" containerID="825a6dd97ad1b2a6cb1b31a1267c4636ac4a7bba986165f4474570890b06976b"
Nov 03 19:52:57.758741 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.758647    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825a6dd97ad1b2a6cb1b31a1267c4636ac4a7bba986165f4474570890b06976b"} err="failed to get container status \"825a6dd97ad1b2a6cb1b31a1267c4636ac4a7bba986165f4474570890b06976b\": rpc error: code = NotFound desc = could not find container \"825a6dd97ad1b2a6cb1b31a1267c4636ac4a7bba986165f4474570890b06976b\": container with ID starting with 825a6dd97ad1b2a6cb1b31a1267c4636ac4a7bba986165f4474570890b06976b not found: ID does not exist"
Nov 03 19:52:57.758741 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.758673    5237 scope.go:122] "RemoveContainer" containerID="cfd97b8b32ad10cb8935759925e24ac4782719a685c08cfd998e432968cbb223"
Nov 03 19:52:57.843897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.797738    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"]
Nov 03 19:52:57.898002 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.897931    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-435bd8-746649f7c7-8jr68"]
Nov 03 19:52:57.898002 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.897980    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"]
Nov 03 19:52:57.898002 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.898004    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-763a69-5fd5d5947c-7p227"]
Nov 03 19:52:57.904293 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:57.903869    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:58.939861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:58.939720    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:59.950860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:59.946759    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3882ca5a-6385-41e0-892c-21f22d58ddc9" path="/var/lib/kubelet/pods/3882ca5a-6385-41e0-892c-21f22d58ddc9/volumes"
Nov 03 19:52:59.976973 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:52:59.976926    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae" path="/var/lib/kubelet/pods/8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae/volumes"
Nov 03 19:53:00.154548 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:00.154477    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:00.962858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:00.958503    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:01.132914 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:01.132875    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:01.353603 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:01.353568    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"]
Nov 03 19:53:01.353810 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:01.353784    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6" podUID="68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" containerName="istio-proxy" containerID="cri-o://f12187738d358a21d868f3d8e9c782a398744f7cc12321fb5c36190767c7a186" gracePeriod=30
Nov 03 19:53:01.403111 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:01.403076    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:01.864860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:01.863034    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:02.537607 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:02.537569    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:03.002198 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:03.002072    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:03.148061 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:03.148025    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:03.346135 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:03.346101    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"]
Nov 03 19:53:03.346621 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:03.346587    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj" podUID="9fea1109-4466-401f-8de1-3c9fae530f70" containerName="istio-proxy" containerID="cri-o://26fd557315a82ab01b4184b1a866811260e9298aa88ea92b84d0cf9f5c926cee" gracePeriod=30
Nov 03 19:53:03.354858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:03.354614    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:03.357684 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:03.357656    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"]
Nov 03 19:53:03.358943 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:03.358063    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q" podUID="ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" containerName="istio-proxy" containerID="cri-o://9befdf34124b75d112ad5fd1b1501c36aa97a372c830cd4ffa78eedbb9acb680" gracePeriod=30
Nov 03 19:53:03.363536 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:03.363512    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"]
Nov 03 19:53:03.363979 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:03.363936    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw" podUID="704a8596-0c3b-403f-b847-7ce2256eb13d" containerName="istio-proxy" containerID="cri-o://7cb82223e95f22f85a75a443e366aae304db9aa20ff14c35fd298b2bf01a9870" gracePeriod=30
Nov 03 19:53:03.507606 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:03.507571    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:03.739448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:03.739354    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:03.959897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:03.954080    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"]
Nov 03 19:53:03.959897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:03.954302    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm" podUID="e6a4dcf8-18ac-443b-b78b-014a4e38ce34" containerName="istio-proxy" containerID="cri-o://ee6f02cf22caf9e58e6e26855329365b5065a44226c9605f0b0dc53f6c118e8f" gracePeriod=30
Nov 03 19:53:03.976553 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:03.976518    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"]
Nov 03 19:53:03.976800 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:03.976772    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" podUID="bd104015-ca9a-4193-bf07-f26376a66366" containerName="istio-proxy" containerID="cri-o://4222219fb4ec3ee20b71b4e5f53a26d0b6ee4e14f050a2b9be9302feba69f1b4" gracePeriod=30
Nov 03 19:53:04.021345 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:04.021246    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:04.241510 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:04.241466    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:53:04.241675 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:04.241537    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj" podUID="9fea1109-4466-401f-8de1-3c9fae530f70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:53:04.278546 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:04.278369    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:53:04.278546 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:04.278432    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm" podUID="e6a4dcf8-18ac-443b-b78b-014a4e38ce34" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:53:04.448182 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:04.448061    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:05.047953 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:05.047822    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:06.047290 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.047251    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:06.274629 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.274591    5237 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:53:06.275029 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.274665    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" podUID="bd104015-ca9a-4193-bf07-f26376a66366" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:53:06.543411 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.543379    5237 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:53:06.543557 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.543449    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q" podUID="ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:53:06.631626 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.631568    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:53:06.684085 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.684052    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-envoy\") pod \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") "
Nov 03 19:53:06.684260 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.684102    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-token\" (UniqueName: \"kubernetes.io/projected/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-token\") pod \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") "
Nov 03 19:53:06.684260 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.684168    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-workload-socket\") pod \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") "
Nov 03 19:53:06.684260 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.684244    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-workload-certs\") pod \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") "
Nov 03 19:53:06.684409 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.684278    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-credential-socket\") pod \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") "
Nov 03 19:53:06.684409 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.684349    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-kube-api-access-j49mz\" (UniqueName: \"kubernetes.io/projected/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-kube-api-access-j49mz\") pod \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") "
Nov 03 19:53:06.684409 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.684379    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istiod-ca-cert\") pod \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") "
Nov 03 19:53:06.684560 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.684414    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-data\") pod \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") "
Nov 03 19:53:06.684560 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.684442    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-podinfo\") pod \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\" (UID: \"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a\") "
Nov 03 19:53:06.684672 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.684552    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-workload-certs" pod "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" (UID: "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:06.684771 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.684532    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-workload-socket" pod "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" (UID: "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:06.684864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.684768    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-data" pod "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" (UID: "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:06.684930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.684871    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istiod-ca-cert" pod "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" (UID: "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:06.685045 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.685021    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:06.685118 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.685052    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:06.685118 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.685068    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:06.685118 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.685081    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:06.685293 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.685160    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-credential-socket" pod "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" (UID: "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:06.686473 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.686446    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-envoy" pod "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" (UID: "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:06.686797 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.686777    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-kube-api-access-j49mz" pod "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" (UID: "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a"). InnerVolumeSpecName "kube-api-access-j49mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:06.686888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.686807    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-token" pod "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" (UID: "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:06.686969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.686947    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-podinfo" pod "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" (UID: "68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:53:06.786585 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.786551    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:06.786585 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.786582    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j49mz\" (UniqueName: \"kubernetes.io/projected/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-kube-api-access-j49mz\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:06.786585 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.786593    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:06.786824 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.786602    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:06.786824 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.786611    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:06.890896 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:06.890797    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:07.041067 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.041021    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:07.102395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.102363    5237 generic.go:356] "Generic (PLEG): container finished" podID="68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" containerID="f12187738d358a21d868f3d8e9c782a398744f7cc12321fb5c36190767c7a186" exitCode=0
Nov 03 19:53:07.102578 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.102410    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"
Nov 03 19:53:07.102578 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.102449    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6" event={"ID":"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a","Type":"ContainerDied","Data":"f12187738d358a21d868f3d8e9c782a398744f7cc12321fb5c36190767c7a186"}
Nov 03 19:53:07.102578 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.102506    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6" event={"ID":"68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a","Type":"ContainerDied","Data":"66e7c91815ce8fd34b9c97c6ab84917c2f4725cb4425632ed2772d12daa5051f"}
Nov 03 19:53:07.102578 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.102535    5237 scope.go:122] "RemoveContainer" containerID="f12187738d358a21d868f3d8e9c782a398744f7cc12321fb5c36190767c7a186"
Nov 03 19:53:07.112904 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.112882    5237 scope.go:122] "RemoveContainer" containerID="7ec9b427e11b45b9ede06afc80f5789fcf3b67c99b2780c46cb5e4d248623663"
Nov 03 19:53:07.122318 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.122294    5237 scope.go:122] "RemoveContainer" containerID="f12187738d358a21d868f3d8e9c782a398744f7cc12321fb5c36190767c7a186"
Nov 03 19:53:07.122608 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:53:07.122588    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f12187738d358a21d868f3d8e9c782a398744f7cc12321fb5c36190767c7a186\": container with ID starting with f12187738d358a21d868f3d8e9c782a398744f7cc12321fb5c36190767c7a186 not found: ID does not exist" containerID="f12187738d358a21d868f3d8e9c782a398744f7cc12321fb5c36190767c7a186"
Nov 03 19:53:07.122678 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.122619    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f12187738d358a21d868f3d8e9c782a398744f7cc12321fb5c36190767c7a186"} err="failed to get container status \"f12187738d358a21d868f3d8e9c782a398744f7cc12321fb5c36190767c7a186\": rpc error: code = NotFound desc = could not find container \"f12187738d358a21d868f3d8e9c782a398744f7cc12321fb5c36190767c7a186\": container with ID starting with f12187738d358a21d868f3d8e9c782a398744f7cc12321fb5c36190767c7a186 not found: ID does not exist"
Nov 03 19:53:07.122678 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.122637    5237 scope.go:122] "RemoveContainer" containerID="7ec9b427e11b45b9ede06afc80f5789fcf3b67c99b2780c46cb5e4d248623663"
Nov 03 19:53:07.122933 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:53:07.122913    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ec9b427e11b45b9ede06afc80f5789fcf3b67c99b2780c46cb5e4d248623663\": container with ID starting with 7ec9b427e11b45b9ede06afc80f5789fcf3b67c99b2780c46cb5e4d248623663 not found: ID does not exist" containerID="7ec9b427e11b45b9ede06afc80f5789fcf3b67c99b2780c46cb5e4d248623663"
Nov 03 19:53:07.123003 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.122942    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ec9b427e11b45b9ede06afc80f5789fcf3b67c99b2780c46cb5e4d248623663"} err="failed to get container status \"7ec9b427e11b45b9ede06afc80f5789fcf3b67c99b2780c46cb5e4d248623663\": rpc error: code = NotFound desc = could not find container \"7ec9b427e11b45b9ede06afc80f5789fcf3b67c99b2780c46cb5e4d248623663\": container with ID starting with 7ec9b427e11b45b9ede06afc80f5789fcf3b67c99b2780c46cb5e4d248623663 not found: ID does not exist"
Nov 03 19:53:07.133891 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.133866    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"]
Nov 03 19:53:07.141469 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.141416    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-763a69-openshift-default-d9f96b999-7fll6"]
Nov 03 19:53:07.529080 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.528993    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"]
Nov 03 19:53:07.529508 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.529200    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" podUID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" containerName="istio-proxy" containerID="cri-o://a1e7d01aece73653fc1cd8e5ffee735994481db53489c9378869bfba2d50553e" gracePeriod=30
Nov 03 19:53:07.547392 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.547363    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"]
Nov 03 19:53:07.547575 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.547555    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" containerID="cri-o://ebdeea957a9ae3743cc865dc0892088979ccb6a2f363a772e786ef8f52816fe1" gracePeriod=30
Nov 03 19:53:07.622620 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.622574    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:07.875874 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.860694    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" path="/var/lib/kubelet/pods/68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a/volumes"
Nov 03 19:53:07.979994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:07.979956    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:08.114144 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.114066    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:08.264049 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.263932    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:08.419540 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.418392    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:08.662433 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.662403    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:53:08.689956 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.689928    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:53:08.718526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718024    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-workload-certs\") pod \"704a8596-0c3b-403f-b847-7ce2256eb13d\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") "
Nov 03 19:53:08.718526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718074    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-istio-envoy\") pod \"9fea1109-4466-401f-8de1-3c9fae530f70\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") "
Nov 03 19:53:08.718526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718103    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/704a8596-0c3b-403f-b847-7ce2256eb13d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704a8596-0c3b-403f-b847-7ce2256eb13d-istiod-ca-cert\") pod \"704a8596-0c3b-403f-b847-7ce2256eb13d\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") "
Nov 03 19:53:08.718526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718131    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fea1109-4466-401f-8de1-3c9fae530f70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fea1109-4466-401f-8de1-3c9fae530f70-istio-podinfo\") pod \"9fea1109-4466-401f-8de1-3c9fae530f70\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") "
Nov 03 19:53:08.718526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718152    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-workload-socket\") pod \"9fea1109-4466-401f-8de1-3c9fae530f70\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") "
Nov 03 19:53:08.718526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718174    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-token\" (UniqueName: \"kubernetes.io/projected/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-token\") pod \"704a8596-0c3b-403f-b847-7ce2256eb13d\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") "
Nov 03 19:53:08.718526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718194    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fea1109-4466-401f-8de1-3c9fae530f70-istio-token\" (UniqueName: \"kubernetes.io/projected/9fea1109-4466-401f-8de1-3c9fae530f70-istio-token\") pod \"9fea1109-4466-401f-8de1-3c9fae530f70\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") "
Nov 03 19:53:08.718526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718241    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fea1109-4466-401f-8de1-3c9fae530f70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fea1109-4466-401f-8de1-3c9fae530f70-istiod-ca-cert\") pod \"9fea1109-4466-401f-8de1-3c9fae530f70\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") "
Nov 03 19:53:08.718526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718274    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-istio-data\") pod \"9fea1109-4466-401f-8de1-3c9fae530f70\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") "
Nov 03 19:53:08.718526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718291    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-envoy\") pod \"704a8596-0c3b-403f-b847-7ce2256eb13d\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") "
Nov 03 19:53:08.718526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718323    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-workload-socket\") pod \"704a8596-0c3b-403f-b847-7ce2256eb13d\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") "
Nov 03 19:53:08.718526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718396    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-podinfo\") pod \"704a8596-0c3b-403f-b847-7ce2256eb13d\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") "
Nov 03 19:53:08.718526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718416    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-credential-socket\") pod \"704a8596-0c3b-403f-b847-7ce2256eb13d\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") "
Nov 03 19:53:08.718526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718465    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704a8596-0c3b-403f-b847-7ce2256eb13d-kube-api-access-w9c52\" (UniqueName: \"kubernetes.io/projected/704a8596-0c3b-403f-b847-7ce2256eb13d-kube-api-access-w9c52\") pod \"704a8596-0c3b-403f-b847-7ce2256eb13d\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") "
Nov 03 19:53:08.718526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718485    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fea1109-4466-401f-8de1-3c9fae530f70-kube-api-access-7nw9p\" (UniqueName: \"kubernetes.io/projected/9fea1109-4466-401f-8de1-3c9fae530f70-kube-api-access-7nw9p\") pod \"9fea1109-4466-401f-8de1-3c9fae530f70\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") "
Nov 03 19:53:08.718526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718514    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-credential-socket\") pod \"9fea1109-4466-401f-8de1-3c9fae530f70\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") "
Nov 03 19:53:08.719732 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718561    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-data\") pod \"704a8596-0c3b-403f-b847-7ce2256eb13d\" (UID: \"704a8596-0c3b-403f-b847-7ce2256eb13d\") "
Nov 03 19:53:08.719732 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.718599    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-workload-certs\") pod \"9fea1109-4466-401f-8de1-3c9fae530f70\" (UID: \"9fea1109-4466-401f-8de1-3c9fae530f70\") "
Nov 03 19:53:08.719732 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.719314    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-workload-certs" pod "9fea1109-4466-401f-8de1-3c9fae530f70" (UID: "9fea1109-4466-401f-8de1-3c9fae530f70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:08.719732 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.719612    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704a8596-0c3b-403f-b847-7ce2256eb13d-istiod-ca-cert" pod "704a8596-0c3b-403f-b847-7ce2256eb13d" (UID: "704a8596-0c3b-403f-b847-7ce2256eb13d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:08.731333 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.720925    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-workload-certs" pod "704a8596-0c3b-403f-b847-7ce2256eb13d" (UID: "704a8596-0c3b-403f-b847-7ce2256eb13d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:08.731333 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.721558    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-istio-data" pod "9fea1109-4466-401f-8de1-3c9fae530f70" (UID: "9fea1109-4466-401f-8de1-3c9fae530f70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:08.731333 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.722010    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fea1109-4466-401f-8de1-3c9fae530f70-istiod-ca-cert" pod "9fea1109-4466-401f-8de1-3c9fae530f70" (UID: "9fea1109-4466-401f-8de1-3c9fae530f70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:08.731333 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.722419    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fea1109-4466-401f-8de1-3c9fae530f70-istio-podinfo" pod "9fea1109-4466-401f-8de1-3c9fae530f70" (UID: "9fea1109-4466-401f-8de1-3c9fae530f70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:53:08.731333 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.722552    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-credential-socket" pod "9fea1109-4466-401f-8de1-3c9fae530f70" (UID: "9fea1109-4466-401f-8de1-3c9fae530f70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:08.731333 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.722579    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-workload-socket" pod "9fea1109-4466-401f-8de1-3c9fae530f70" (UID: "9fea1109-4466-401f-8de1-3c9fae530f70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:08.731333 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.722649    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-data" pod "704a8596-0c3b-403f-b847-7ce2256eb13d" (UID: "704a8596-0c3b-403f-b847-7ce2256eb13d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:08.731333 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.723109    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-workload-socket" pod "704a8596-0c3b-403f-b847-7ce2256eb13d" (UID: "704a8596-0c3b-403f-b847-7ce2256eb13d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:08.731333 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.723297    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-credential-socket" pod "704a8596-0c3b-403f-b847-7ce2256eb13d" (UID: "704a8596-0c3b-403f-b847-7ce2256eb13d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:08.731333 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.726891    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-istio-envoy" pod "9fea1109-4466-401f-8de1-3c9fae530f70" (UID: "9fea1109-4466-401f-8de1-3c9fae530f70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:08.731333 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.730162    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-token" pod "704a8596-0c3b-403f-b847-7ce2256eb13d" (UID: "704a8596-0c3b-403f-b847-7ce2256eb13d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:08.731333 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.731131    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-podinfo" pod "704a8596-0c3b-403f-b847-7ce2256eb13d" (UID: "704a8596-0c3b-403f-b847-7ce2256eb13d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:53:08.732380 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.732343    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704a8596-0c3b-403f-b847-7ce2256eb13d-kube-api-access-w9c52" pod "704a8596-0c3b-403f-b847-7ce2256eb13d" (UID: "704a8596-0c3b-403f-b847-7ce2256eb13d"). InnerVolumeSpecName "kube-api-access-w9c52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:08.732640 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.732591    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fea1109-4466-401f-8de1-3c9fae530f70-kube-api-access-7nw9p" pod "9fea1109-4466-401f-8de1-3c9fae530f70" (UID: "9fea1109-4466-401f-8de1-3c9fae530f70"). InnerVolumeSpecName "kube-api-access-7nw9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:08.732640 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.732611    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fea1109-4466-401f-8de1-3c9fae530f70-istio-token" pod "9fea1109-4466-401f-8de1-3c9fae530f70" (UID: "9fea1109-4466-401f-8de1-3c9fae530f70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:08.734607 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.734579    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-envoy" pod "704a8596-0c3b-403f-b847-7ce2256eb13d" (UID: "704a8596-0c3b-403f-b847-7ce2256eb13d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:08.820395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820314    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.820395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820343    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.820395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820353    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9c52\" (UniqueName: \"kubernetes.io/projected/704a8596-0c3b-403f-b847-7ce2256eb13d-kube-api-access-w9c52\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.820395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820363    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nw9p\" (UniqueName: \"kubernetes.io/projected/9fea1109-4466-401f-8de1-3c9fae530f70-kube-api-access-7nw9p\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.820395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820372    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.820395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820381    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.820395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820389    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.820395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820397    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.820759 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820407    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.820759 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820416    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704a8596-0c3b-403f-b847-7ce2256eb13d-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.820759 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820424    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fea1109-4466-401f-8de1-3c9fae530f70-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.820759 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820433    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.820759 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820441    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.820759 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820449    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fea1109-4466-401f-8de1-3c9fae530f70-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.820759 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820457    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fea1109-4466-401f-8de1-3c9fae530f70-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.820759 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820464    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fea1109-4466-401f-8de1-3c9fae530f70-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.820759 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820472    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.820759 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.820480    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704a8596-0c3b-403f-b847-7ce2256eb13d-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.886584 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:08.886529    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:09.023501 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.023456    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:09.090004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.089978    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:53:09.125699 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.124292    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-envoy\") pod \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") "
Nov 03 19:53:09.125699 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.124326    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-workload-certs\") pod \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") "
Nov 03 19:53:09.125699 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.124369    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-credential-socket\") pod \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") "
Nov 03 19:53:09.125699 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.124415    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-token\" (UniqueName: \"kubernetes.io/projected/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-token\") pod \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") "
Nov 03 19:53:09.125699 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.124442    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-kube-api-access-g5nwc\" (UniqueName: \"kubernetes.io/projected/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-kube-api-access-g5nwc\") pod \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") "
Nov 03 19:53:09.125699 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.124461    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-workload-socket\") pod \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") "
Nov 03 19:53:09.125699 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.124481    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istiod-ca-cert\") pod \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") "
Nov 03 19:53:09.125699 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.124508    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-podinfo\") pod \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") "
Nov 03 19:53:09.125699 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.124531    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-data\") pod \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\" (UID: \"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b\") "
Nov 03 19:53:09.125699 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.125013    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-data" pod "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" (UID: "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.126589 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.126561    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istiod-ca-cert" pod "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" (UID: "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:09.126888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.126679    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-workload-socket" pod "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" (UID: "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.126888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.126751    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-credential-socket" pod "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" (UID: "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.126888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.126777    5237 generic.go:356] "Generic (PLEG): container finished" podID="e6a4dcf8-18ac-443b-b78b-014a4e38ce34" containerID="ee6f02cf22caf9e58e6e26855329365b5065a44226c9605f0b0dc53f6c118e8f" exitCode=0
Nov 03 19:53:09.126888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.126806    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm" event={"ID":"e6a4dcf8-18ac-443b-b78b-014a4e38ce34","Type":"ContainerDied","Data":"ee6f02cf22caf9e58e6e26855329365b5065a44226c9605f0b0dc53f6c118e8f"}
Nov 03 19:53:09.127107 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.126988    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-workload-certs" pod "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" (UID: "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.128242 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.128204    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-envoy" pod "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" (UID: "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.129361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.128590    5237 generic.go:356] "Generic (PLEG): container finished" podID="9fea1109-4466-401f-8de1-3c9fae530f70" containerID="26fd557315a82ab01b4184b1a866811260e9298aa88ea92b84d0cf9f5c926cee" exitCode=0
Nov 03 19:53:09.129361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.128753    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"
Nov 03 19:53:09.129361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.128806    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj" event={"ID":"9fea1109-4466-401f-8de1-3c9fae530f70","Type":"ContainerDied","Data":"26fd557315a82ab01b4184b1a866811260e9298aa88ea92b84d0cf9f5c926cee"}
Nov 03 19:53:09.129361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.128877    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj" event={"ID":"9fea1109-4466-401f-8de1-3c9fae530f70","Type":"ContainerDied","Data":"7b7caa6101499b58e18d0cc3d2bcb3d4792e8e3455c58758496547eb32e9acf1"}
Nov 03 19:53:09.129361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.128884    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-podinfo" pod "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" (UID: "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:53:09.129361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.128911    5237 scope.go:122] "RemoveContainer" containerID="26fd557315a82ab01b4184b1a866811260e9298aa88ea92b84d0cf9f5c926cee"
Nov 03 19:53:09.129361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.128967    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-kube-api-access-g5nwc" pod "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" (UID: "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b"). InnerVolumeSpecName "kube-api-access-g5nwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.129962 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.129902    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-token" pod "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" (UID: "ad53de78-95ea-4f6b-b6a2-caa3e86bd49b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.130880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.130823    5237 generic.go:356] "Generic (PLEG): container finished" podID="bd104015-ca9a-4193-bf07-f26376a66366" containerID="4222219fb4ec3ee20b71b4e5f53a26d0b6ee4e14f050a2b9be9302feba69f1b4" exitCode=0
Nov 03 19:53:09.130880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.130862    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" event={"ID":"bd104015-ca9a-4193-bf07-f26376a66366","Type":"ContainerDied","Data":"4222219fb4ec3ee20b71b4e5f53a26d0b6ee4e14f050a2b9be9302feba69f1b4"}
Nov 03 19:53:09.132476 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.132451    5237 generic.go:356] "Generic (PLEG): container finished" podID="704a8596-0c3b-403f-b847-7ce2256eb13d" containerID="7cb82223e95f22f85a75a443e366aae304db9aa20ff14c35fd298b2bf01a9870" exitCode=0
Nov 03 19:53:09.132616 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.132597    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw" event={"ID":"704a8596-0c3b-403f-b847-7ce2256eb13d","Type":"ContainerDied","Data":"7cb82223e95f22f85a75a443e366aae304db9aa20ff14c35fd298b2bf01a9870"}
Nov 03 19:53:09.132684 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.132640    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw" event={"ID":"704a8596-0c3b-403f-b847-7ce2256eb13d","Type":"ContainerDied","Data":"bbcffa68573b5fea1af7fe458011a3db6885f512a91b0abd20da8b78cb52aa1e"}
Nov 03 19:53:09.132756 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.132742    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"
Nov 03 19:53:09.145151 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.144399    5237 scope.go:122] "RemoveContainer" containerID="26fd557315a82ab01b4184b1a866811260e9298aa88ea92b84d0cf9f5c926cee"
Nov 03 19:53:09.146951 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:53:09.145294    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26fd557315a82ab01b4184b1a866811260e9298aa88ea92b84d0cf9f5c926cee\": container with ID starting with 26fd557315a82ab01b4184b1a866811260e9298aa88ea92b84d0cf9f5c926cee not found: ID does not exist" containerID="26fd557315a82ab01b4184b1a866811260e9298aa88ea92b84d0cf9f5c926cee"
Nov 03 19:53:09.146951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.145331    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26fd557315a82ab01b4184b1a866811260e9298aa88ea92b84d0cf9f5c926cee"} err="failed to get container status \"26fd557315a82ab01b4184b1a866811260e9298aa88ea92b84d0cf9f5c926cee\": rpc error: code = NotFound desc = could not find container \"26fd557315a82ab01b4184b1a866811260e9298aa88ea92b84d0cf9f5c926cee\": container with ID starting with 26fd557315a82ab01b4184b1a866811260e9298aa88ea92b84d0cf9f5c926cee not found: ID does not exist"
Nov 03 19:53:09.146951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.145356    5237 scope.go:122] "RemoveContainer" containerID="7cb82223e95f22f85a75a443e366aae304db9aa20ff14c35fd298b2bf01a9870"
Nov 03 19:53:09.146951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.146390    5237 generic.go:356] "Generic (PLEG): container finished" podID="ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" containerID="9befdf34124b75d112ad5fd1b1501c36aa97a372c830cd4ffa78eedbb9acb680" exitCode=0
Nov 03 19:53:09.146951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.146467    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q" event={"ID":"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b","Type":"ContainerDied","Data":"9befdf34124b75d112ad5fd1b1501c36aa97a372c830cd4ffa78eedbb9acb680"}
Nov 03 19:53:09.146951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.146497    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q" event={"ID":"ad53de78-95ea-4f6b-b6a2-caa3e86bd49b","Type":"ContainerDied","Data":"35b540413ed684c047894a18c7c08566caa48661460c5aec43f135470f34c672"}
Nov 03 19:53:09.146951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.146550    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"
Nov 03 19:53:09.168406 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.168386    5237 scope.go:122] "RemoveContainer" containerID="7cb82223e95f22f85a75a443e366aae304db9aa20ff14c35fd298b2bf01a9870"
Nov 03 19:53:09.168814 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:53:09.168789    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb82223e95f22f85a75a443e366aae304db9aa20ff14c35fd298b2bf01a9870\": container with ID starting with 7cb82223e95f22f85a75a443e366aae304db9aa20ff14c35fd298b2bf01a9870 not found: ID does not exist" containerID="7cb82223e95f22f85a75a443e366aae304db9aa20ff14c35fd298b2bf01a9870"
Nov 03 19:53:09.168949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.168822    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb82223e95f22f85a75a443e366aae304db9aa20ff14c35fd298b2bf01a9870"} err="failed to get container status \"7cb82223e95f22f85a75a443e366aae304db9aa20ff14c35fd298b2bf01a9870\": rpc error: code = NotFound desc = could not find container \"7cb82223e95f22f85a75a443e366aae304db9aa20ff14c35fd298b2bf01a9870\": container with ID starting with 7cb82223e95f22f85a75a443e366aae304db9aa20ff14c35fd298b2bf01a9870 not found: ID does not exist"
Nov 03 19:53:09.168949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.168859    5237 scope.go:122] "RemoveContainer" containerID="9befdf34124b75d112ad5fd1b1501c36aa97a372c830cd4ffa78eedbb9acb680"
Nov 03 19:53:09.182308 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.182208    5237 scope.go:122] "RemoveContainer" containerID="9befdf34124b75d112ad5fd1b1501c36aa97a372c830cd4ffa78eedbb9acb680"
Nov 03 19:53:09.182777 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:53:09.182705    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9befdf34124b75d112ad5fd1b1501c36aa97a372c830cd4ffa78eedbb9acb680\": container with ID starting with 9befdf34124b75d112ad5fd1b1501c36aa97a372c830cd4ffa78eedbb9acb680 not found: ID does not exist" containerID="9befdf34124b75d112ad5fd1b1501c36aa97a372c830cd4ffa78eedbb9acb680"
Nov 03 19:53:09.182777 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.182741    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9befdf34124b75d112ad5fd1b1501c36aa97a372c830cd4ffa78eedbb9acb680"} err="failed to get container status \"9befdf34124b75d112ad5fd1b1501c36aa97a372c830cd4ffa78eedbb9acb680\": rpc error: code = NotFound desc = could not find container \"9befdf34124b75d112ad5fd1b1501c36aa97a372c830cd4ffa78eedbb9acb680\": container with ID starting with 9befdf34124b75d112ad5fd1b1501c36aa97a372c830cd4ffa78eedbb9acb680 not found: ID does not exist"
Nov 03 19:53:09.222576 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.222543    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"]
Nov 03 19:53:09.224447 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.224420    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:53:09.225361 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.225341    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.225448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.225364    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.225448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.225376    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.225448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.225385    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.225448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.225394    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.225448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.225403    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.225448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.225412    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5nwc\" (UniqueName: \"kubernetes.io/projected/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-kube-api-access-g5nwc\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.225448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.225422    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.225448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.225431    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.232434 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.232399    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-c8c2q"]
Nov 03 19:53:09.248157 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.248121    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"]
Nov 03 19:53:09.255491 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.255463    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-7b2kw"]
Nov 03 19:53:09.272130 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.270072    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"]
Nov 03 19:53:09.274453 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.274427    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-khfqj"]
Nov 03 19:53:09.281213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.281191    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:53:09.326373 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.326338    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd104015-ca9a-4193-bf07-f26376a66366-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd104015-ca9a-4193-bf07-f26376a66366-istiod-ca-cert\") pod \"bd104015-ca9a-4193-bf07-f26376a66366\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") "
Nov 03 19:53:09.326549 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.326392    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-istio-envoy\") pod \"bd104015-ca9a-4193-bf07-f26376a66366\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") "
Nov 03 19:53:09.326549 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.326432    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd104015-ca9a-4193-bf07-f26376a66366-kube-api-access-dchcz\" (UniqueName: \"kubernetes.io/projected/bd104015-ca9a-4193-bf07-f26376a66366-kube-api-access-dchcz\") pod \"bd104015-ca9a-4193-bf07-f26376a66366\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") "
Nov 03 19:53:09.326644 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.326569    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-kube-api-access-4k6gn\" (UniqueName: \"kubernetes.io/projected/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-kube-api-access-4k6gn\") pod \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") "
Nov 03 19:53:09.326644 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.326605    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-credential-socket\") pod \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") "
Nov 03 19:53:09.326644 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.326634    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-istio-data\") pod \"bd104015-ca9a-4193-bf07-f26376a66366\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") "
Nov 03 19:53:09.326781 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.326671    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-workload-socket\") pod \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") "
Nov 03 19:53:09.326781 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.326697    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-workload-certs\") pod \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") "
Nov 03 19:53:09.326781 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.326728    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd104015-ca9a-4193-bf07-f26376a66366-istio-token\" (UniqueName: \"kubernetes.io/projected/bd104015-ca9a-4193-bf07-f26376a66366-istio-token\") pod \"bd104015-ca9a-4193-bf07-f26376a66366\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") "
Nov 03 19:53:09.326781 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.326757    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-token\" (UniqueName: \"kubernetes.io/projected/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-token\") pod \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") "
Nov 03 19:53:09.326972 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.326786    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd104015-ca9a-4193-bf07-f26376a66366-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd104015-ca9a-4193-bf07-f26376a66366-istio-podinfo\") pod \"bd104015-ca9a-4193-bf07-f26376a66366\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") "
Nov 03 19:53:09.326972 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.326816    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-workload-socket\") pod \"bd104015-ca9a-4193-bf07-f26376a66366\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") "
Nov 03 19:53:09.326972 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.326913    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-envoy\") pod \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") "
Nov 03 19:53:09.327073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.326975    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-podinfo\") pod \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") "
Nov 03 19:53:09.327073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.327005    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-credential-socket\") pod \"bd104015-ca9a-4193-bf07-f26376a66366\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") "
Nov 03 19:53:09.327073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.327050    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-data\") pod \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") "
Nov 03 19:53:09.327217 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.327084    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-workload-certs\") pod \"bd104015-ca9a-4193-bf07-f26376a66366\" (UID: \"bd104015-ca9a-4193-bf07-f26376a66366\") "
Nov 03 19:53:09.327217 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.327170    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istiod-ca-cert\") pod \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\" (UID: \"e6a4dcf8-18ac-443b-b78b-014a4e38ce34\") "
Nov 03 19:53:09.327620 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.327588    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-workload-socket" pod "bd104015-ca9a-4193-bf07-f26376a66366" (UID: "bd104015-ca9a-4193-bf07-f26376a66366"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.327949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.327924    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istiod-ca-cert" pod "e6a4dcf8-18ac-443b-b78b-014a4e38ce34" (UID: "e6a4dcf8-18ac-443b-b78b-014a4e38ce34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:09.328376 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.328354    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-credential-socket" pod "bd104015-ca9a-4193-bf07-f26376a66366" (UID: "bd104015-ca9a-4193-bf07-f26376a66366"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.328692 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.328671    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-data" pod "e6a4dcf8-18ac-443b-b78b-014a4e38ce34" (UID: "e6a4dcf8-18ac-443b-b78b-014a4e38ce34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.329010 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.328991    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-workload-certs" pod "bd104015-ca9a-4193-bf07-f26376a66366" (UID: "bd104015-ca9a-4193-bf07-f26376a66366"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.329301 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.329273    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-workload-certs" pod "e6a4dcf8-18ac-443b-b78b-014a4e38ce34" (UID: "e6a4dcf8-18ac-443b-b78b-014a4e38ce34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.329495 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.329472    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-workload-socket" pod "e6a4dcf8-18ac-443b-b78b-014a4e38ce34" (UID: "e6a4dcf8-18ac-443b-b78b-014a4e38ce34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.329932 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.329685    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-credential-socket" pod "e6a4dcf8-18ac-443b-b78b-014a4e38ce34" (UID: "e6a4dcf8-18ac-443b-b78b-014a4e38ce34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.330323 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.330065    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-envoy" pod "e6a4dcf8-18ac-443b-b78b-014a4e38ce34" (UID: "e6a4dcf8-18ac-443b-b78b-014a4e38ce34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.330323 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.330090    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-istio-data" pod "bd104015-ca9a-4193-bf07-f26376a66366" (UID: "bd104015-ca9a-4193-bf07-f26376a66366"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.330716 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.330660    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd104015-ca9a-4193-bf07-f26376a66366-istiod-ca-cert" pod "bd104015-ca9a-4193-bf07-f26376a66366" (UID: "bd104015-ca9a-4193-bf07-f26376a66366"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:09.331785 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.331726    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd104015-ca9a-4193-bf07-f26376a66366-istio-token" pod "bd104015-ca9a-4193-bf07-f26376a66366" (UID: "bd104015-ca9a-4193-bf07-f26376a66366"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.332263 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.332069    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-token" pod "e6a4dcf8-18ac-443b-b78b-014a4e38ce34" (UID: "e6a4dcf8-18ac-443b-b78b-014a4e38ce34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.333363 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.333338    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-podinfo" pod "e6a4dcf8-18ac-443b-b78b-014a4e38ce34" (UID: "e6a4dcf8-18ac-443b-b78b-014a4e38ce34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:53:09.333797 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.333763    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-kube-api-access-4k6gn" pod "e6a4dcf8-18ac-443b-b78b-014a4e38ce34" (UID: "e6a4dcf8-18ac-443b-b78b-014a4e38ce34"). InnerVolumeSpecName "kube-api-access-4k6gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.334010 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.333987    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-istio-envoy" pod "bd104015-ca9a-4193-bf07-f26376a66366" (UID: "bd104015-ca9a-4193-bf07-f26376a66366"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.334941 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.334889    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd104015-ca9a-4193-bf07-f26376a66366-kube-api-access-dchcz" pod "bd104015-ca9a-4193-bf07-f26376a66366" (UID: "bd104015-ca9a-4193-bf07-f26376a66366"). InnerVolumeSpecName "kube-api-access-dchcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.335391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.335370    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd104015-ca9a-4193-bf07-f26376a66366-istio-podinfo" pod "bd104015-ca9a-4193-bf07-f26376a66366" (UID: "bd104015-ca9a-4193-bf07-f26376a66366"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:53:09.429034 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429003    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k6gn\" (UniqueName: \"kubernetes.io/projected/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-kube-api-access-4k6gn\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.429034 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429032    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.429201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429042    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.429201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429051    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.429201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429060    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.429201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429068    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd104015-ca9a-4193-bf07-f26376a66366-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.429201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429076    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.429201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429084    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd104015-ca9a-4193-bf07-f26376a66366-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.429201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429092    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.429201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429103    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.429201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429112    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.429201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429120    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.429201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429128    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.429201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429136    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.429201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429143    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a4dcf8-18ac-443b-b78b-014a4e38ce34-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.429201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429151    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd104015-ca9a-4193-bf07-f26376a66366-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.429201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429159    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd104015-ca9a-4193-bf07-f26376a66366-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.429201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.429168    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dchcz\" (UniqueName: \"kubernetes.io/projected/bd104015-ca9a-4193-bf07-f26376a66366-kube-api-access-dchcz\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.822790 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.822750    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:09.864293 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.864255    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="704a8596-0c3b-403f-b847-7ce2256eb13d" path="/var/lib/kubelet/pods/704a8596-0c3b-403f-b847-7ce2256eb13d/volumes"
Nov 03 19:53:09.864972 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.864953    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fea1109-4466-401f-8de1-3c9fae530f70" path="/var/lib/kubelet/pods/9fea1109-4466-401f-8de1-3c9fae530f70/volumes"
Nov 03 19:53:09.865527 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:09.865511    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" path="/var/lib/kubelet/pods/ad53de78-95ea-4f6b-b6a2-caa3e86bd49b/volumes"
Nov 03 19:53:10.154402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:10.154321    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"
Nov 03 19:53:10.154402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:10.154352    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm" event={"ID":"e6a4dcf8-18ac-443b-b78b-014a4e38ce34","Type":"ContainerDied","Data":"b36774d446631d5b4ed10bbdc15eee6a09bbbe96de2361d3c6c7f9b5bd4d500f"}
Nov 03 19:53:10.154644 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:10.154426    5237 scope.go:122] "RemoveContainer" containerID="ee6f02cf22caf9e58e6e26855329365b5065a44226c9605f0b0dc53f6c118e8f"
Nov 03 19:53:10.157261 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:10.157237    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"
Nov 03 19:53:10.158388 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:10.157746    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg" event={"ID":"bd104015-ca9a-4193-bf07-f26376a66366","Type":"ContainerDied","Data":"141c27d75637a581748c364f49215e632a34a2a3f6b557c4ddf2219ebff9b0d1"}
Nov 03 19:53:10.165817 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:10.165801    5237 scope.go:122] "RemoveContainer" containerID="4222219fb4ec3ee20b71b4e5f53a26d0b6ee4e14f050a2b9be9302feba69f1b4"
Nov 03 19:53:10.177845 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:10.177811    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"]
Nov 03 19:53:10.184142 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:10.184113    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-258fg"]
Nov 03 19:53:10.202370 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:10.202341    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"]
Nov 03 19:53:10.209951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:10.209918    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-k2rmm"]
Nov 03 19:53:10.561935 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:10.561898    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:10.765168 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:10.765127    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:10.906123 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:10.906019    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:11.827171 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:11.827138    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:11.859664 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:11.859634    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd104015-ca9a-4193-bf07-f26376a66366" path="/var/lib/kubelet/pods/bd104015-ca9a-4193-bf07-f26376a66366/volumes"
Nov 03 19:53:11.860138 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:11.860123    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a4dcf8-18ac-443b-b78b-014a4e38ce34" path="/var/lib/kubelet/pods/e6a4dcf8-18ac-443b-b78b-014a4e38ce34/volumes"
Nov 03 19:53:12.024569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.024536    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"]
Nov 03 19:53:12.140422 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.140338    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:12.273102 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.273068    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:12.485485 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.485395    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"]
Nov 03 19:53:12.485675 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.485646    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24" podUID="98b3d450-df16-4c35-8095-4d0640c7a422" containerName="maas-api" containerID="cri-o://5681ee443c0c567d40acc1b7926be71ffa54ce037b8dfd29ef4fc78311e15898" gracePeriod=30
Nov 03 19:53:12.497169 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.496872    5237 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:53:12.497169 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.496936    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:53:12.568222 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.568188    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:12.723856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.723228    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:12.856302 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.856138    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:12.884735 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.884702    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:53:12.910553 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.910528    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:53:12.924862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.924303    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:53:12.980987 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.980954    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98b3d450-df16-4c35-8095-4d0640c7a422-metrics-certs\" (UniqueName: \"kubernetes.io/secret/98b3d450-df16-4c35-8095-4d0640c7a422-metrics-certs\") pod \"98b3d450-df16-4c35-8095-4d0640c7a422\" (UID: \"98b3d450-df16-4c35-8095-4d0640c7a422\") "
Nov 03 19:53:12.981176 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981026    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-workload-socket\") pod \"a48f6c83-ae97-4927-9d94-2d17fd679536\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") "
Nov 03 19:53:12.981176 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981064    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-data\") pod \"a48f6c83-ae97-4927-9d94-2d17fd679536\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") "
Nov 03 19:53:12.981176 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981099    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98b3d450-df16-4c35-8095-4d0640c7a422-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b3d450-df16-4c35-8095-4d0640c7a422-openshift-service-ca\") pod \"98b3d450-df16-4c35-8095-4d0640c7a422\" (UID: \"98b3d450-df16-4c35-8095-4d0640c7a422\") "
Nov 03 19:53:12.981176 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981129    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-token\" (UniqueName: \"kubernetes.io/projected/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-token\") pod \"a48f6c83-ae97-4927-9d94-2d17fd679536\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") "
Nov 03 19:53:12.981176 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981169    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98b3d450-df16-4c35-8095-4d0640c7a422-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98b3d450-df16-4c35-8095-4d0640c7a422-maas-api-tls\") pod \"98b3d450-df16-4c35-8095-4d0640c7a422\" (UID: \"98b3d450-df16-4c35-8095-4d0640c7a422\") "
Nov 03 19:53:12.981475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981198    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-token\" (UniqueName: \"kubernetes.io/projected/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-token\") pod \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") "
Nov 03 19:53:12.981475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981243    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-workload-socket\") pod \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") "
Nov 03 19:53:12.981475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981277    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-envoy\") pod \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") "
Nov 03 19:53:12.981475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981332    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-envoy\") pod \"a48f6c83-ae97-4927-9d94-2d17fd679536\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") "
Nov 03 19:53:12.981475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981332    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-workload-socket" pod "a48f6c83-ae97-4927-9d94-2d17fd679536" (UID: "a48f6c83-ae97-4927-9d94-2d17fd679536"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:12.981475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981386    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-data\") pod \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") "
Nov 03 19:53:12.981475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981390    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-data" pod "a48f6c83-ae97-4927-9d94-2d17fd679536" (UID: "a48f6c83-ae97-4927-9d94-2d17fd679536"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:12.981475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981434    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-workload-certs\") pod \"a48f6c83-ae97-4927-9d94-2d17fd679536\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") "
Nov 03 19:53:12.981475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981467    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-kube-api-access-4ggjh\" (UniqueName: \"kubernetes.io/projected/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-kube-api-access-4ggjh\") pod \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") "
Nov 03 19:53:12.981961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981521    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-podinfo\") pod \"a48f6c83-ae97-4927-9d94-2d17fd679536\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") "
Nov 03 19:53:12.981961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981548    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istiod-ca-cert\") pod \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") "
Nov 03 19:53:12.981961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981575    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-workload-certs\") pod \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") "
Nov 03 19:53:12.981961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981611    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-credential-socket\") pod \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") "
Nov 03 19:53:12.981961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981658    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-credential-socket\") pod \"a48f6c83-ae97-4927-9d94-2d17fd679536\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") "
Nov 03 19:53:12.981961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981688    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98b3d450-df16-4c35-8095-4d0640c7a422-kube-api-access-fj48n\" (UniqueName: \"kubernetes.io/projected/98b3d450-df16-4c35-8095-4d0640c7a422-kube-api-access-fj48n\") pod \"98b3d450-df16-4c35-8095-4d0640c7a422\" (UID: \"98b3d450-df16-4c35-8095-4d0640c7a422\") "
Nov 03 19:53:12.981961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981702    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-workload-socket" pod "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" (UID: "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:12.981961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981718    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48f6c83-ae97-4927-9d94-2d17fd679536-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a48f6c83-ae97-4927-9d94-2d17fd679536-istiod-ca-cert\") pod \"a48f6c83-ae97-4927-9d94-2d17fd679536\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") "
Nov 03 19:53:12.981961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981756    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-podinfo\") pod \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\" (UID: \"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f\") "
Nov 03 19:53:12.981961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.981799    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48f6c83-ae97-4927-9d94-2d17fd679536-kube-api-access-jbwkl\" (UniqueName: \"kubernetes.io/projected/a48f6c83-ae97-4927-9d94-2d17fd679536-kube-api-access-jbwkl\") pod \"a48f6c83-ae97-4927-9d94-2d17fd679536\" (UID: \"a48f6c83-ae97-4927-9d94-2d17fd679536\") "
Nov 03 19:53:12.982469 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.982270    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:12.982469 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.982294    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:12.982469 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.982309    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:12.983409 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.982892    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-workload-certs" pod "a48f6c83-ae97-4927-9d94-2d17fd679536" (UID: "a48f6c83-ae97-4927-9d94-2d17fd679536"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:12.983409 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.983098    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-data" pod "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" (UID: "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:12.983409 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.983266    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istiod-ca-cert" pod "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" (UID: "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:12.983409 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.983277    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-credential-socket" pod "a48f6c83-ae97-4927-9d94-2d17fd679536" (UID: "a48f6c83-ae97-4927-9d94-2d17fd679536"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:12.983409 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.983394    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-workload-certs" pod "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" (UID: "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:12.983749 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.983545    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-credential-socket" pod "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" (UID: "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:12.985772 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.985734    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b3d450-df16-4c35-8095-4d0640c7a422-maas-api-tls" pod "98b3d450-df16-4c35-8095-4d0640c7a422" (UID: "98b3d450-df16-4c35-8095-4d0640c7a422"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:53:12.986783 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.986760    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b3d450-df16-4c35-8095-4d0640c7a422-kube-api-access-fj48n" pod "98b3d450-df16-4c35-8095-4d0640c7a422" (UID: "98b3d450-df16-4c35-8095-4d0640c7a422"). InnerVolumeSpecName "kube-api-access-fj48n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:12.987135 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.987118    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-podinfo" pod "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" (UID: "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:53:12.988021 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.987992    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b3d450-df16-4c35-8095-4d0640c7a422-openshift-service-ca" pod "98b3d450-df16-4c35-8095-4d0640c7a422" (UID: "98b3d450-df16-4c35-8095-4d0640c7a422"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:12.988420 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.988380    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48f6c83-ae97-4927-9d94-2d17fd679536-istiod-ca-cert" pod "a48f6c83-ae97-4927-9d94-2d17fd679536" (UID: "a48f6c83-ae97-4927-9d94-2d17fd679536"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:12.988741 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.988715    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:12.993805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.989560    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-podinfo" pod "a48f6c83-ae97-4927-9d94-2d17fd679536" (UID: "a48f6c83-ae97-4927-9d94-2d17fd679536"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:53:12.993805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.989599    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-token" pod "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" (UID: "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:12.993805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.989706    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-envoy" pod "a48f6c83-ae97-4927-9d94-2d17fd679536" (UID: "a48f6c83-ae97-4927-9d94-2d17fd679536"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:12.993805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.989711    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-token" pod "a48f6c83-ae97-4927-9d94-2d17fd679536" (UID: "a48f6c83-ae97-4927-9d94-2d17fd679536"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:12.993805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.989830    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-envoy" pod "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" (UID: "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:12.993805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.989988    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-kube-api-access-4ggjh" pod "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" (UID: "e2acf71f-02e6-4f7f-aa4a-4daa381bc69f"). InnerVolumeSpecName "kube-api-access-4ggjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:12.993805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.991547    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b3d450-df16-4c35-8095-4d0640c7a422-metrics-certs" pod "98b3d450-df16-4c35-8095-4d0640c7a422" (UID: "98b3d450-df16-4c35-8095-4d0640c7a422"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:53:12.993805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:12.991564    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48f6c83-ae97-4927-9d94-2d17fd679536-kube-api-access-jbwkl" pod "a48f6c83-ae97-4927-9d94-2d17fd679536" (UID: "a48f6c83-ae97-4927-9d94-2d17fd679536"). InnerVolumeSpecName "kube-api-access-jbwkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:13.083715 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083677    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbwkl\" (UniqueName: \"kubernetes.io/projected/a48f6c83-ae97-4927-9d94-2d17fd679536-kube-api-access-jbwkl\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.083715 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083709    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98b3d450-df16-4c35-8095-4d0640c7a422-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.083715 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083721    5237 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b3d450-df16-4c35-8095-4d0640c7a422-openshift-service-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.084213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083730    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.084213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083739    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98b3d450-df16-4c35-8095-4d0640c7a422-maas-api-tls\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.084213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083749    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.084213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083757    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.084213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083767    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.084213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083777    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.084213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083785    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.084213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083793    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ggjh\" (UniqueName: \"kubernetes.io/projected/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-kube-api-access-4ggjh\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.084213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083803    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a48f6c83-ae97-4927-9d94-2d17fd679536-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.084213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083813    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.084213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083826    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.084213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083852    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.084213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083861    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48f6c83-ae97-4927-9d94-2d17fd679536-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.084213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083871    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj48n\" (UniqueName: \"kubernetes.io/projected/98b3d450-df16-4c35-8095-4d0640c7a422-kube-api-access-fj48n\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.084213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083879    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a48f6c83-ae97-4927-9d94-2d17fd679536-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.084213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.083887    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.183404 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.183368    5237 generic.go:356] "Generic (PLEG): container finished" podID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerID="ebdeea957a9ae3743cc865dc0892088979ccb6a2f363a772e786ef8f52816fe1" exitCode=0
Nov 03 19:53:13.183586 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.183416    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"
Nov 03 19:53:13.183586 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.183455    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" event={"ID":"a48f6c83-ae97-4927-9d94-2d17fd679536","Type":"ContainerDied","Data":"ebdeea957a9ae3743cc865dc0892088979ccb6a2f363a772e786ef8f52816fe1"}
Nov 03 19:53:13.183586 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.183515    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq" event={"ID":"a48f6c83-ae97-4927-9d94-2d17fd679536","Type":"ContainerDied","Data":"d4f7adaa7118592118a73af2f73d4d17d71f4bbfe3f4f019fc0790f111adff92"}
Nov 03 19:53:13.183586 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.183544    5237 scope.go:122] "RemoveContainer" containerID="ebdeea957a9ae3743cc865dc0892088979ccb6a2f363a772e786ef8f52816fe1"
Nov 03 19:53:13.185363 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.185338    5237 generic.go:356] "Generic (PLEG): container finished" podID="98b3d450-df16-4c35-8095-4d0640c7a422" containerID="5681ee443c0c567d40acc1b7926be71ffa54ce037b8dfd29ef4fc78311e15898" exitCode=0
Nov 03 19:53:13.185363 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.185360    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"
Nov 03 19:53:13.185541 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.185382    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24" event={"ID":"98b3d450-df16-4c35-8095-4d0640c7a422","Type":"ContainerDied","Data":"5681ee443c0c567d40acc1b7926be71ffa54ce037b8dfd29ef4fc78311e15898"}
Nov 03 19:53:13.185541 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.185435    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24" event={"ID":"98b3d450-df16-4c35-8095-4d0640c7a422","Type":"ContainerDied","Data":"df0f6d396e961b7ca688e6fafe1299e4a4cf2c5b7816549bce6573f976fb7d8f"}
Nov 03 19:53:13.188913 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.188886    5237 generic.go:356] "Generic (PLEG): container finished" podID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" containerID="a1e7d01aece73653fc1cd8e5ffee735994481db53489c9378869bfba2d50553e" exitCode=0
Nov 03 19:53:13.189031 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.188935    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"
Nov 03 19:53:13.189121 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.188938    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" event={"ID":"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f","Type":"ContainerDied","Data":"a1e7d01aece73653fc1cd8e5ffee735994481db53489c9378869bfba2d50553e"}
Nov 03 19:53:13.189190 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.189152    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw" event={"ID":"e2acf71f-02e6-4f7f-aa4a-4daa381bc69f","Type":"ContainerDied","Data":"245b1733a50762672fc16d85e80287d0186d562c42af19173cb1294199e0beb4"}
Nov 03 19:53:13.197254 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.197172    5237 scope.go:122] "RemoveContainer" containerID="ebdeea957a9ae3743cc865dc0892088979ccb6a2f363a772e786ef8f52816fe1"
Nov 03 19:53:13.197480 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:53:13.197459    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebdeea957a9ae3743cc865dc0892088979ccb6a2f363a772e786ef8f52816fe1\": container with ID starting with ebdeea957a9ae3743cc865dc0892088979ccb6a2f363a772e786ef8f52816fe1 not found: ID does not exist" containerID="ebdeea957a9ae3743cc865dc0892088979ccb6a2f363a772e786ef8f52816fe1"
Nov 03 19:53:13.197545 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.197488    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebdeea957a9ae3743cc865dc0892088979ccb6a2f363a772e786ef8f52816fe1"} err="failed to get container status \"ebdeea957a9ae3743cc865dc0892088979ccb6a2f363a772e786ef8f52816fe1\": rpc error: code = NotFound desc = could not find container \"ebdeea957a9ae3743cc865dc0892088979ccb6a2f363a772e786ef8f52816fe1\": container with ID starting with ebdeea957a9ae3743cc865dc0892088979ccb6a2f363a772e786ef8f52816fe1 not found: ID does not exist"
Nov 03 19:53:13.197545 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.197503    5237 scope.go:122] "RemoveContainer" containerID="5681ee443c0c567d40acc1b7926be71ffa54ce037b8dfd29ef4fc78311e15898"
Nov 03 19:53:13.206155 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.206134    5237 scope.go:122] "RemoveContainer" containerID="5681ee443c0c567d40acc1b7926be71ffa54ce037b8dfd29ef4fc78311e15898"
Nov 03 19:53:13.206463 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:53:13.206442    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5681ee443c0c567d40acc1b7926be71ffa54ce037b8dfd29ef4fc78311e15898\": container with ID starting with 5681ee443c0c567d40acc1b7926be71ffa54ce037b8dfd29ef4fc78311e15898 not found: ID does not exist" containerID="5681ee443c0c567d40acc1b7926be71ffa54ce037b8dfd29ef4fc78311e15898"
Nov 03 19:53:13.206524 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.206469    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5681ee443c0c567d40acc1b7926be71ffa54ce037b8dfd29ef4fc78311e15898"} err="failed to get container status \"5681ee443c0c567d40acc1b7926be71ffa54ce037b8dfd29ef4fc78311e15898\": rpc error: code = NotFound desc = could not find container \"5681ee443c0c567d40acc1b7926be71ffa54ce037b8dfd29ef4fc78311e15898\": container with ID starting with 5681ee443c0c567d40acc1b7926be71ffa54ce037b8dfd29ef4fc78311e15898 not found: ID does not exist"
Nov 03 19:53:13.206524 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.206485    5237 scope.go:122] "RemoveContainer" containerID="a1e7d01aece73653fc1cd8e5ffee735994481db53489c9378869bfba2d50553e"
Nov 03 19:53:13.212030 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.212007    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"]
Nov 03 19:53:13.214891 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.214870    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-ww9gq"]
Nov 03 19:53:13.215187 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.215151    5237 scope.go:122] "RemoveContainer" containerID="a1e7d01aece73653fc1cd8e5ffee735994481db53489c9378869bfba2d50553e"
Nov 03 19:53:13.215466 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:53:13.215448    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1e7d01aece73653fc1cd8e5ffee735994481db53489c9378869bfba2d50553e\": container with ID starting with a1e7d01aece73653fc1cd8e5ffee735994481db53489c9378869bfba2d50553e not found: ID does not exist" containerID="a1e7d01aece73653fc1cd8e5ffee735994481db53489c9378869bfba2d50553e"
Nov 03 19:53:13.215511 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.215473    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1e7d01aece73653fc1cd8e5ffee735994481db53489c9378869bfba2d50553e"} err="failed to get container status \"a1e7d01aece73653fc1cd8e5ffee735994481db53489c9378869bfba2d50553e\": rpc error: code = NotFound desc = could not find container \"a1e7d01aece73653fc1cd8e5ffee735994481db53489c9378869bfba2d50553e\": container with ID starting with a1e7d01aece73653fc1cd8e5ffee735994481db53489c9378869bfba2d50553e not found: ID does not exist"
Nov 03 19:53:13.233942 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.233917    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"]
Nov 03 19:53:13.249692 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.249656    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-r2shw"]
Nov 03 19:53:13.262356 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:53:13.262324    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:53:13.264963 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.264936    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"]
Nov 03 19:53:13.273220 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.273194    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-64454a-795b6fb8ff-cgn24"]
Nov 03 19:53:13.422675 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.422591    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:13.870893 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.869183    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:13.875030 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.874997    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98b3d450-df16-4c35-8095-4d0640c7a422" path="/var/lib/kubelet/pods/98b3d450-df16-4c35-8095-4d0640c7a422/volumes"
Nov 03 19:53:13.875592 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.875570    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" path="/var/lib/kubelet/pods/a48f6c83-ae97-4927-9d94-2d17fd679536/volumes"
Nov 03 19:53:13.876213 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:13.876193    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" path="/var/lib/kubelet/pods/e2acf71f-02e6-4f7f-aa4a-4daa381bc69f/volumes"
Nov 03 19:53:14.551864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:14.551806    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:15.292858 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:15.291159    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:16.251395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:16.251146    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:16.399918 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:16.399875    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:16.712890 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:16.712814    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:16.863118 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:16.863076    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:17.016965 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:17.002608    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:17.033532 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:17.025506    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j" podUID="8d820e94-7b5a-48c6-8351-a45ea23de377" containerName="payload-pre-processing" containerID="cri-o://47880980c933540001ef68c5ce806efcd1c6481541c9dbb56962b7ea289f3b22" gracePeriod=25
Nov 03 19:53:17.225799 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:17.225758    5237 generic.go:356] "Generic (PLEG): container finished" podID="8d820e94-7b5a-48c6-8351-a45ea23de377" containerID="47880980c933540001ef68c5ce806efcd1c6481541c9dbb56962b7ea289f3b22" exitCode=0
Nov 03 19:53:17.226001 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:17.225944    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j" event={"ID":"8d820e94-7b5a-48c6-8351-a45ea23de377","Type":"ContainerDied","Data":"47880980c933540001ef68c5ce806efcd1c6481541c9dbb56962b7ea289f3b22"}
Nov 03 19:53:17.288445 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:17.288421    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"
Nov 03 19:53:17.331377 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:17.330359    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d820e94-7b5a-48c6-8351-a45ea23de377-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d820e94-7b5a-48c6-8351-a45ea23de377-plugins-config-volume\") pod \"8d820e94-7b5a-48c6-8351-a45ea23de377\" (UID: \"8d820e94-7b5a-48c6-8351-a45ea23de377\") "
Nov 03 19:53:17.331377 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:17.330498    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d820e94-7b5a-48c6-8351-a45ea23de377-kube-api-access-qfhdk\" (UniqueName: \"kubernetes.io/projected/8d820e94-7b5a-48c6-8351-a45ea23de377-kube-api-access-qfhdk\") pod \"8d820e94-7b5a-48c6-8351-a45ea23de377\" (UID: \"8d820e94-7b5a-48c6-8351-a45ea23de377\") "
Nov 03 19:53:17.331894 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:17.331758    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d820e94-7b5a-48c6-8351-a45ea23de377-plugins-config-volume" pod "8d820e94-7b5a-48c6-8351-a45ea23de377" (UID: "8d820e94-7b5a-48c6-8351-a45ea23de377"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:17.332512 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:17.332478    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:17.334946 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:17.334666    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d820e94-7b5a-48c6-8351-a45ea23de377-kube-api-access-qfhdk" pod "8d820e94-7b5a-48c6-8351-a45ea23de377" (UID: "8d820e94-7b5a-48c6-8351-a45ea23de377"). InnerVolumeSpecName "kube-api-access-qfhdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:17.432290 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:17.432251    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfhdk\" (UniqueName: \"kubernetes.io/projected/8d820e94-7b5a-48c6-8351-a45ea23de377-kube-api-access-qfhdk\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:17.432290 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:17.432281    5237 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d820e94-7b5a-48c6-8351-a45ea23de377-plugins-config-volume\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:17.533161 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:17.533080    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:18.035016 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:18.034976    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:18.232859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:18.232770    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"
Nov 03 19:53:18.232859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:18.232774    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j" event={"ID":"8d820e94-7b5a-48c6-8351-a45ea23de377","Type":"ContainerDied","Data":"a23d183ec77f69ad634758d9a1fda837790fd80def9752eafd7830703aa4a5f7"}
Nov 03 19:53:18.233094 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:18.232864    5237 scope.go:122] "RemoveContainer" containerID="47880980c933540001ef68c5ce806efcd1c6481541c9dbb56962b7ea289f3b22"
Nov 03 19:53:18.253350 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:18.253314    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"]
Nov 03 19:53:18.258534 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:18.258506    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-64454a-85c79c67dc-whr6j"]
Nov 03 19:53:18.869868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:18.861074    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:19.255475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:19.255389    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:19.648286 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:19.648252    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:19.858817 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:19.858785    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d820e94-7b5a-48c6-8351-a45ea23de377" path="/var/lib/kubelet/pods/8d820e94-7b5a-48c6-8351-a45ea23de377/volumes"
Nov 03 19:53:20.451650 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:20.449847    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:20.623772 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:20.623735    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:20.845028 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:20.844990    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:21.192012 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:21.191886    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:21.406304 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:21.406262    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:21.591949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:21.591913    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:22.411662 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:22.411183    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:23.154861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:23.154656    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:23.969648 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:23.969610    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:24.642060 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:24.642022    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:25.351899 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:25.351796    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:26.073153 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:26.073116    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:26.215335 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:26.215297    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:26.510447 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:26.510338    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"]
Nov 03 19:53:26.529735 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:26.529701    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"]
Nov 03 19:53:26.698877 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:26.698807    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:26.828927 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:26.828880    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:27.409524 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:27.409472    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:28.135349 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:28.135310    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:28.795497 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:28.795463    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:29.672641 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:29.672605    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:30.362182 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:30.362145    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:30.733885 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:30.733758    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:31.212030 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:31.211996    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:31.350094 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:31.350059    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:31.818157 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:31.818114    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:31.971095 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:31.971060    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:32.529041 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:32.529001    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:41.531049 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:41.530982    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main" containerID="cri-o://aa1524641693abd488840945ccefd3e4bd7989823c7c184606cb3efaa31e7b63" gracePeriod=45
Nov 03 19:53:41.552303 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:41.552263    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main" containerID="cri-o://9144af6a893056cd8d0099f9fe9a68f086263711f652b51151c8ad83d7a41b34" gracePeriod=45
Nov 03 19:53:41.819542 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:41.819514    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:53:41.844542 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:41.844501    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:53:42.011623 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.011582    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-tls-certs\") pod \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") "
Nov 03 19:53:42.011787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.011634    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-dshm\") pod \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") "
Nov 03 19:53:42.011787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.011696    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-kube-api-access-5jbb9\" (UniqueName: \"kubernetes.io/projected/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-kube-api-access-5jbb9\") pod \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") "
Nov 03 19:53:42.011787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.011724    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-home\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-home\") pod \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") "
Nov 03 19:53:42.011787 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.011766    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-tmp-dir\") pod \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") "
Nov 03 19:53:42.012087 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.011807    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-home\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-home\") pod \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") "
Nov 03 19:53:42.012087 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.011880    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-tls-certs\") pod \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") "
Nov 03 19:53:42.012087 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.011913    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-tmp-dir\") pod \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") "
Nov 03 19:53:42.012087 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.011946    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-model-cache\") pod \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") "
Nov 03 19:53:42.012087 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.011980    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-kube-api-access-hc4wr\" (UniqueName: \"kubernetes.io/projected/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-kube-api-access-hc4wr\") pod \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") "
Nov 03 19:53:42.012087 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.012018    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-home" pod "a39a1945-62e0-42c7-9b2a-d81d2ee042d6" (UID: "a39a1945-62e0-42c7-9b2a-d81d2ee042d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:42.012087 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.012030    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-model-cache\") pod \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\" (UID: \"1a61b39a-3f50-4ff7-88fe-dc8f851db36b\") "
Nov 03 19:53:42.012475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.012102    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-dshm\") pod \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\" (UID: \"a39a1945-62e0-42c7-9b2a-d81d2ee042d6\") "
Nov 03 19:53:42.012475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.012165    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-home" pod "1a61b39a-3f50-4ff7-88fe-dc8f851db36b" (UID: "1a61b39a-3f50-4ff7-88fe-dc8f851db36b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:42.012475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.012254    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-model-cache" pod "1a61b39a-3f50-4ff7-88fe-dc8f851db36b" (UID: "1a61b39a-3f50-4ff7-88fe-dc8f851db36b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:42.012475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.012351    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-tmp-dir" pod "a39a1945-62e0-42c7-9b2a-d81d2ee042d6" (UID: "a39a1945-62e0-42c7-9b2a-d81d2ee042d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:42.012475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.012192    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-tmp-dir" pod "1a61b39a-3f50-4ff7-88fe-dc8f851db36b" (UID: "1a61b39a-3f50-4ff7-88fe-dc8f851db36b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:42.012475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.012466    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-model-cache" pod "a39a1945-62e0-42c7-9b2a-d81d2ee042d6" (UID: "a39a1945-62e0-42c7-9b2a-d81d2ee042d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:42.012751 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.012493    5237 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-home\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:42.012751 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.012514    5237 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-tmp-dir\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:42.012751 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.012529    5237 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-home\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:42.012751 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.012542    5237 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-tmp-dir\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:42.012751 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.012554    5237 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-model-cache\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:42.014532 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.014509    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-tls-certs" pod "1a61b39a-3f50-4ff7-88fe-dc8f851db36b" (UID: "1a61b39a-3f50-4ff7-88fe-dc8f851db36b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:53:42.014632 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.014560    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-dshm" pod "1a61b39a-3f50-4ff7-88fe-dc8f851db36b" (UID: "1a61b39a-3f50-4ff7-88fe-dc8f851db36b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:42.014632 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.014584    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-kube-api-access-5jbb9" pod "1a61b39a-3f50-4ff7-88fe-dc8f851db36b" (UID: "1a61b39a-3f50-4ff7-88fe-dc8f851db36b"). InnerVolumeSpecName "kube-api-access-5jbb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:42.014736 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.014665    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-dshm" pod "a39a1945-62e0-42c7-9b2a-d81d2ee042d6" (UID: "a39a1945-62e0-42c7-9b2a-d81d2ee042d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:42.014736 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.014664    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-tls-certs" pod "a39a1945-62e0-42c7-9b2a-d81d2ee042d6" (UID: "a39a1945-62e0-42c7-9b2a-d81d2ee042d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:53:42.014736 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.014701    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-kube-api-access-hc4wr" pod "a39a1945-62e0-42c7-9b2a-d81d2ee042d6" (UID: "a39a1945-62e0-42c7-9b2a-d81d2ee042d6"). InnerVolumeSpecName "kube-api-access-hc4wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:42.113624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.113537    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jbb9\" (UniqueName: \"kubernetes.io/projected/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-kube-api-access-5jbb9\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:42.113624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.113572    5237 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-tls-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:42.113624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.113585    5237 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-model-cache\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:42.113624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.113595    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc4wr\" (UniqueName: \"kubernetes.io/projected/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-kube-api-access-hc4wr\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:42.113624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.113603    5237 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a39a1945-62e0-42c7-9b2a-d81d2ee042d6-dshm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:42.113624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.113612    5237 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-tls-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:42.113624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.113620    5237 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a61b39a-3f50-4ff7-88fe-dc8f851db36b-dshm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:42.355033 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.354995    5237 generic.go:356] "Generic (PLEG): container finished" podID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerID="9144af6a893056cd8d0099f9fe9a68f086263711f652b51151c8ad83d7a41b34" exitCode=0
Nov 03 19:53:42.355200 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.355040    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"
Nov 03 19:53:42.355200 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.355076    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" event={"ID":"a39a1945-62e0-42c7-9b2a-d81d2ee042d6","Type":"ContainerDied","Data":"9144af6a893056cd8d0099f9fe9a68f086263711f652b51151c8ad83d7a41b34"}
Nov 03 19:53:42.355200 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.355118    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94" event={"ID":"a39a1945-62e0-42c7-9b2a-d81d2ee042d6","Type":"ContainerDied","Data":"dc5d06f7065f5884b9b83557b3098f5e7156e42c12b612052e13426073583846"}
Nov 03 19:53:42.355200 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.355137    5237 scope.go:122] "RemoveContainer" containerID="9144af6a893056cd8d0099f9fe9a68f086263711f652b51151c8ad83d7a41b34"
Nov 03 19:53:42.356870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.356480    5237 generic.go:356] "Generic (PLEG): container finished" podID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerID="aa1524641693abd488840945ccefd3e4bd7989823c7c184606cb3efaa31e7b63" exitCode=0
Nov 03 19:53:42.356870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.356513    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" event={"ID":"1a61b39a-3f50-4ff7-88fe-dc8f851db36b","Type":"ContainerDied","Data":"aa1524641693abd488840945ccefd3e4bd7989823c7c184606cb3efaa31e7b63"}
Nov 03 19:53:42.356870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.356559    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v" event={"ID":"1a61b39a-3f50-4ff7-88fe-dc8f851db36b","Type":"ContainerDied","Data":"2c930c6bcb0d93b213d8bcd08ea1f0e7bc222ed4031c5c2d4cfd7d410974d530"}
Nov 03 19:53:42.356870 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.356526    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"
Nov 03 19:53:42.363515 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.363486    5237 scope.go:122] "RemoveContainer" containerID="9144af6a893056cd8d0099f9fe9a68f086263711f652b51151c8ad83d7a41b34"
Nov 03 19:53:42.363849 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:53:42.363787    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9144af6a893056cd8d0099f9fe9a68f086263711f652b51151c8ad83d7a41b34\": container with ID starting with 9144af6a893056cd8d0099f9fe9a68f086263711f652b51151c8ad83d7a41b34 not found: ID does not exist" containerID="9144af6a893056cd8d0099f9fe9a68f086263711f652b51151c8ad83d7a41b34"
Nov 03 19:53:42.363849 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.363817    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9144af6a893056cd8d0099f9fe9a68f086263711f652b51151c8ad83d7a41b34"} err="failed to get container status \"9144af6a893056cd8d0099f9fe9a68f086263711f652b51151c8ad83d7a41b34\": rpc error: code = NotFound desc = could not find container \"9144af6a893056cd8d0099f9fe9a68f086263711f652b51151c8ad83d7a41b34\": container with ID starting with 9144af6a893056cd8d0099f9fe9a68f086263711f652b51151c8ad83d7a41b34 not found: ID does not exist"
Nov 03 19:53:42.363849 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.363847    5237 scope.go:122] "RemoveContainer" containerID="aa1524641693abd488840945ccefd3e4bd7989823c7c184606cb3efaa31e7b63"
Nov 03 19:53:42.372064 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.372044    5237 scope.go:122] "RemoveContainer" containerID="aa1524641693abd488840945ccefd3e4bd7989823c7c184606cb3efaa31e7b63"
Nov 03 19:53:42.372319 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:53:42.372300    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1524641693abd488840945ccefd3e4bd7989823c7c184606cb3efaa31e7b63\": container with ID starting with aa1524641693abd488840945ccefd3e4bd7989823c7c184606cb3efaa31e7b63 not found: ID does not exist" containerID="aa1524641693abd488840945ccefd3e4bd7989823c7c184606cb3efaa31e7b63"
Nov 03 19:53:42.372386 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.372326    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1524641693abd488840945ccefd3e4bd7989823c7c184606cb3efaa31e7b63"} err="failed to get container status \"aa1524641693abd488840945ccefd3e4bd7989823c7c184606cb3efaa31e7b63\": rpc error: code = NotFound desc = could not find container \"aa1524641693abd488840945ccefd3e4bd7989823c7c184606cb3efaa31e7b63\": container with ID starting with aa1524641693abd488840945ccefd3e4bd7989823c7c184606cb3efaa31e7b63 not found: ID does not exist"
Nov 03 19:53:42.382059 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.382032    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"]
Nov 03 19:53:42.386709 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.386688    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d8a4ea/simulator-w4-d8a4ea-kserve-95cfc6d5d-sgj94"]
Nov 03 19:53:42.398626 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.398603    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"]
Nov 03 19:53:42.401204 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.401183    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d8a4ea/distinct-w4-d8a4ea-kserve-59c5598dc8-vfr4v"]
Nov 03 19:53:42.923961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:42.923927    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:43.059659 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:43.059620    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:43.254666 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:53:43.254590    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:53:43.858379 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:43.858345    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" path="/var/lib/kubelet/pods/1a61b39a-3f50-4ff7-88fe-dc8f851db36b/volumes"
Nov 03 19:53:43.858767 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:43.858753    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" path="/var/lib/kubelet/pods/a39a1945-62e0-42c7-9b2a-d81d2ee042d6/volumes"
Nov 03 19:53:48.005145 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.005103    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"]
Nov 03 19:53:48.005704 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.005506    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f" podUID="ac52ab6c-a555-494b-b6b9-654d2d60c8e8" containerName="maas-api" containerID="cri-o://a67dd756a8ba0b77313a9f5ef547801fd397e51b00ce17f326436f3949ddbf7e" gracePeriod=30
Nov 03 19:53:48.085908 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.085870    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:48.225170 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.225128    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:48.270109 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.270016    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:53:48.368963 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.368926    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-kube-api-access-2489t\" (UniqueName: \"kubernetes.io/projected/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-kube-api-access-2489t\") pod \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\" (UID: \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\") "
Nov 03 19:53:48.369196 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.369024    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-openshift-service-ca\") pod \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\" (UID: \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\") "
Nov 03 19:53:48.369196 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.369078    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-metrics-certs\") pod \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\" (UID: \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\") "
Nov 03 19:53:48.369196 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.369151    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-maas-api-tls\") pod \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\" (UID: \"ac52ab6c-a555-494b-b6b9-654d2d60c8e8\") "
Nov 03 19:53:48.369419 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.369385    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-openshift-service-ca" pod "ac52ab6c-a555-494b-b6b9-654d2d60c8e8" (UID: "ac52ab6c-a555-494b-b6b9-654d2d60c8e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:48.372030 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.372000    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-maas-api-tls" pod "ac52ab6c-a555-494b-b6b9-654d2d60c8e8" (UID: "ac52ab6c-a555-494b-b6b9-654d2d60c8e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:53:48.372197 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.372100    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-metrics-certs" pod "ac52ab6c-a555-494b-b6b9-654d2d60c8e8" (UID: "ac52ab6c-a555-494b-b6b9-654d2d60c8e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:53:48.372270 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.372197    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-kube-api-access-2489t" pod "ac52ab6c-a555-494b-b6b9-654d2d60c8e8" (UID: "ac52ab6c-a555-494b-b6b9-654d2d60c8e8"). InnerVolumeSpecName "kube-api-access-2489t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:48.388470 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.388442    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:48.424076 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.424041    5237 generic.go:356] "Generic (PLEG): container finished" podID="ac52ab6c-a555-494b-b6b9-654d2d60c8e8" containerID="a67dd756a8ba0b77313a9f5ef547801fd397e51b00ce17f326436f3949ddbf7e" exitCode=0
Nov 03 19:53:48.424271 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.424091    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f" event={"ID":"ac52ab6c-a555-494b-b6b9-654d2d60c8e8","Type":"ContainerDied","Data":"a67dd756a8ba0b77313a9f5ef547801fd397e51b00ce17f326436f3949ddbf7e"}
Nov 03 19:53:48.424271 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.424134    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"
Nov 03 19:53:48.424271 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.424148    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f" event={"ID":"ac52ab6c-a555-494b-b6b9-654d2d60c8e8","Type":"ContainerDied","Data":"3595e531fc8cfe65a02dedd54cce8cc843d33552312c29defc37a23a3be0a172"}
Nov 03 19:53:48.424408 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.424159    5237 scope.go:122] "RemoveContainer" containerID="a67dd756a8ba0b77313a9f5ef547801fd397e51b00ce17f326436f3949ddbf7e"
Nov 03 19:53:48.433740 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.433720    5237 scope.go:122] "RemoveContainer" containerID="a67dd756a8ba0b77313a9f5ef547801fd397e51b00ce17f326436f3949ddbf7e"
Nov 03 19:53:48.434201 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:53:48.434180    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a67dd756a8ba0b77313a9f5ef547801fd397e51b00ce17f326436f3949ddbf7e\": container with ID starting with a67dd756a8ba0b77313a9f5ef547801fd397e51b00ce17f326436f3949ddbf7e not found: ID does not exist" containerID="a67dd756a8ba0b77313a9f5ef547801fd397e51b00ce17f326436f3949ddbf7e"
Nov 03 19:53:48.434282 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.434214    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67dd756a8ba0b77313a9f5ef547801fd397e51b00ce17f326436f3949ddbf7e"} err="failed to get container status \"a67dd756a8ba0b77313a9f5ef547801fd397e51b00ce17f326436f3949ddbf7e\": rpc error: code = NotFound desc = could not find container \"a67dd756a8ba0b77313a9f5ef547801fd397e51b00ce17f326436f3949ddbf7e\": container with ID starting with a67dd756a8ba0b77313a9f5ef547801fd397e51b00ce17f326436f3949ddbf7e not found: ID does not exist"
Nov 03 19:53:48.451692 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.451651    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"]
Nov 03 19:53:48.453469 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.453439    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d8a4ea-79cf9cf798-zq76f"]
Nov 03 19:53:48.470571 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.470537    5237 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-openshift-service-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:48.470571 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.470566    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:48.470571 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.470577    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-maas-api-tls\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:48.470780 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.470585    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2489t\" (UniqueName: \"kubernetes.io/projected/ac52ab6c-a555-494b-b6b9-654d2d60c8e8-kube-api-access-2489t\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:48.811649 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:48.811608    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:49.858647 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:49.858620    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac52ab6c-a555-494b-b6b9-654d2d60c8e8" path="/var/lib/kubelet/pods/ac52ab6c-a555-494b-b6b9-654d2d60c8e8/volumes"
Nov 03 19:53:53.442895 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:53.442857    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:53.575228 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:53.575189    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:54.866572 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:54.866529    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:55.006789 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:55.006755    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:55.209557 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:55.205498    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"]
Nov 03 19:53:55.209557 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:55.205748    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc" podUID="e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" containerName="istio-proxy" containerID="cri-o://0a8ce2fb5a7ef6fbfb1dceb94852294f49490865af064c4cb1c6e59702abdaaa" gracePeriod=30
Nov 03 19:53:55.213149 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:55.213121    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"]
Nov 03 19:53:55.213687 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:55.213656    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9" podUID="7ca1424b-adf4-42f6-8165-44af258b508c" containerName="istio-proxy" containerID="cri-o://863cc691deb31ce822c416bf6f606e67b0889175f0f4e6ea7c48f6039d709292" gracePeriod=30
Nov 03 19:53:55.214425 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:55.214330    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"]
Nov 03 19:53:55.214546 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:55.214501    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp" podUID="6ff03678-8a48-4bdb-83c5-4941236f45bc" containerName="istio-proxy" containerID="cri-o://11fa0d117b7eb1bfd38ec5cb6e216ace2d7d87e5c89c67d0400546bb1ea1c78a" gracePeriod=30
Nov 03 19:53:55.312270 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:55.309531    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:58.329581 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:58.329540    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-694f8bb88d-22dtv"]
Nov 03 19:53:58.330032 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:58.329795    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-694f8bb88d-22dtv" podUID="9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd" containerName="manager" containerID="cri-o://37d885d26711fb23c48aaad791c3841aa8d7706502786aede42d8a36631f817f" gracePeriod=30
Nov 03 19:53:58.485821 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:58.485789    5237 generic.go:356] "Generic (PLEG): container finished" podID="9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd" containerID="37d885d26711fb23c48aaad791c3841aa8d7706502786aede42d8a36631f817f" exitCode=0
Nov 03 19:53:58.485991 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:58.485851    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694f8bb88d-22dtv" event={"ID":"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd","Type":"ContainerDied","Data":"37d885d26711fb23c48aaad791c3841aa8d7706502786aede42d8a36631f817f"}
Nov 03 19:53:58.582148 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:58.582092    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694f8bb88d-22dtv"
Nov 03 19:53:58.669033 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:58.668993    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-metrics-certs\") pod \"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd\" (UID: \"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd\") "
Nov 03 19:53:58.669208 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:58.669100    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-webhook-cert\") pod \"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd\" (UID: \"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd\") "
Nov 03 19:53:58.669208 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:58.669167    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-kube-api-access-zjnm9\" (UniqueName: \"kubernetes.io/projected/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-kube-api-access-zjnm9\") pod \"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd\" (UID: \"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd\") "
Nov 03 19:53:58.671229 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:58.671200    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-webhook-cert" pod "9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd" (UID: "9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:53:58.671338 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:58.671256    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-metrics-certs" pod "9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd" (UID: "9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:53:58.671338 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:58.671275    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-kube-api-access-zjnm9" pod "9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd" (UID: "9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd"). InnerVolumeSpecName "kube-api-access-zjnm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:58.770319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:58.770277    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:58.770319 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:58.770312    5237 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-webhook-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:58.770533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:58.770330    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjnm9\" (UniqueName: \"kubernetes.io/projected/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd-kube-api-access-zjnm9\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:59.493086 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:59.493059    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694f8bb88d-22dtv"
Nov 03 19:53:59.493550 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:59.493052    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694f8bb88d-22dtv" event={"ID":"9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd","Type":"ContainerDied","Data":"2c5e7f026b79aea0767d644ca4a3ad31e54c122cd7ee79798ce7b33f895dd83b"}
Nov 03 19:53:59.493550 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:59.493250    5237 scope.go:122] "RemoveContainer" containerID="37d885d26711fb23c48aaad791c3841aa8d7706502786aede42d8a36631f817f"
Nov 03 19:53:59.522736 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:59.522707    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-694f8bb88d-22dtv"]
Nov 03 19:53:59.527089 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:59.527062    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-694f8bb88d-22dtv"]
Nov 03 19:53:59.859255 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:53:59.859211    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd" path="/var/lib/kubelet/pods/9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd/volumes"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.368409    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-694f8bb88d-vrmx4"]
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.368955    5237 cpu_manager.go:414] "containerMap: removing container" podUID="0398a799-4e11-4f8e-913f-8a42574c0ae9" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.368976    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0398a799-4e11-4f8e-913f-8a42574c0ae9" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.368996    5237 cpu_manager.go:414] "containerMap: removing container" podUID="3882ca5a-6385-41e0-892c-21f22d58ddc9" containerName="payload-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369004    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3882ca5a-6385-41e0-892c-21f22d58ddc9" containerName="payload-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369021    5237 cpu_manager.go:414] "containerMap: removing container" podUID="f6b14de1-629b-4f1a-b35c-cd23b30eff51" containerName="payload-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369030    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b14de1-629b-4f1a-b35c-cd23b30eff51" containerName="payload-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369044    5237 cpu_manager.go:414] "containerMap: removing container" podUID="704a8596-0c3b-403f-b847-7ce2256eb13d" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369052    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704a8596-0c3b-403f-b847-7ce2256eb13d" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369063    5237 cpu_manager.go:414] "containerMap: removing container" podUID="9fea1109-4466-401f-8de1-3c9fae530f70" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369070    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fea1109-4466-401f-8de1-3c9fae530f70" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369081    5237 cpu_manager.go:414] "containerMap: removing container" podUID="98b3d450-df16-4c35-8095-4d0640c7a422" containerName="maas-api"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369088    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98b3d450-df16-4c35-8095-4d0640c7a422" containerName="maas-api"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369101    5237 cpu_manager.go:414] "containerMap: removing container" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369110    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369121    5237 cpu_manager.go:414] "containerMap: removing container" podUID="a4d0eae8-11ce-483c-97f7-e0aef82ed54b" containerName="payload-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369130    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d0eae8-11ce-483c-97f7-e0aef82ed54b" containerName="payload-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369142    5237 cpu_manager.go:414] "containerMap: removing container" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369149    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369161    5237 cpu_manager.go:414] "containerMap: removing container" podUID="5fdb8431-30cd-4023-94e9-47f32412e2e7" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369168    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fdb8431-30cd-4023-94e9-47f32412e2e7" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369179    5237 cpu_manager.go:414] "containerMap: removing container" podUID="e0e9e459-f64b-44b6-a217-56d4e11feada" containerName="payload-pre-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369190    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e9e459-f64b-44b6-a217-56d4e11feada" containerName="payload-pre-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369203    5237 cpu_manager.go:414] "containerMap: removing container" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369212    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369221    5237 cpu_manager.go:414] "containerMap: removing container" podUID="e63a1c5a-826e-487e-b3eb-eab4278b8049" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369228    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63a1c5a-826e-487e-b3eb-eab4278b8049" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369240    5237 cpu_manager.go:414] "containerMap: removing container" podUID="fe7148f3-a2df-43bb-bf04-d5a0fddc1a13" containerName="payload-pre-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369247    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe7148f3-a2df-43bb-bf04-d5a0fddc1a13" containerName="payload-pre-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369261    5237 cpu_manager.go:414] "containerMap: removing container" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369270    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369281    5237 cpu_manager.go:414] "containerMap: removing container" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369288    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369298    5237 cpu_manager.go:414] "containerMap: removing container" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369305    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369314    5237 cpu_manager.go:414] "containerMap: removing container" podUID="bd104015-ca9a-4193-bf07-f26376a66366" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369322    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd104015-ca9a-4193-bf07-f26376a66366" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369334    5237 cpu_manager.go:414] "containerMap: removing container" podUID="93bee0fc-7374-4419-ac8a-07745bdc3a24" containerName="maas-api"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369341    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93bee0fc-7374-4419-ac8a-07745bdc3a24" containerName="maas-api"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369352    5237 cpu_manager.go:414] "containerMap: removing container" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369359    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369371    5237 cpu_manager.go:414] "containerMap: removing container" podUID="e56d7264-7d35-4dd8-946b-c1ab4d40e941" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369377    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56d7264-7d35-4dd8-946b-c1ab4d40e941" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369389    5237 cpu_manager.go:414] "containerMap: removing container" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369396    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369409    5237 cpu_manager.go:414] "containerMap: removing container" podUID="84914ee8-d996-409e-a212-d29119cc5de8" containerName="payload-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369416    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84914ee8-d996-409e-a212-d29119cc5de8" containerName="payload-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369428    5237 cpu_manager.go:414] "containerMap: removing container" podUID="92e6fabb-7410-46bc-a67f-40d9116f02eb" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369435    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e6fabb-7410-46bc-a67f-40d9116f02eb" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369449    5237 cpu_manager.go:414] "containerMap: removing container" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369456    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369468    5237 cpu_manager.go:414] "containerMap: removing container" podUID="941d6f5f-47cd-439e-9463-ecc633799b05" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369475    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941d6f5f-47cd-439e-9463-ecc633799b05" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369487    5237 cpu_manager.go:414] "containerMap: removing container" podUID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369494    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369510    5237 cpu_manager.go:414] "containerMap: removing container" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369518    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369528    5237 cpu_manager.go:414] "containerMap: removing container" podUID="9145fea2-1fed-40d6-82e4-2f79350f9d8d" containerName="maas-api"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369536    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9145fea2-1fed-40d6-82e4-2f79350f9d8d" containerName="maas-api"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369546    5237 cpu_manager.go:414] "containerMap: removing container" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369553    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369563    5237 cpu_manager.go:414] "containerMap: removing container" podUID="9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd" containerName="manager"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369571    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd" containerName="manager"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369581    5237 cpu_manager.go:414] "containerMap: removing container" podUID="6037ccaa-48d9-450c-8726-213ae6b76500" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369588    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6037ccaa-48d9-450c-8726-213ae6b76500" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369600    5237 cpu_manager.go:414] "containerMap: removing container" podUID="8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae" containerName="maas-api"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369607    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae" containerName="maas-api"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369617    5237 cpu_manager.go:414] "containerMap: removing container" podUID="ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369624    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369634    5237 cpu_manager.go:414] "containerMap: removing container" podUID="8d820e94-7b5a-48c6-8351-a45ea23de377" containerName="payload-pre-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369642    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d820e94-7b5a-48c6-8351-a45ea23de377" containerName="payload-pre-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369653    5237 cpu_manager.go:414] "containerMap: removing container" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369660    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369670    5237 cpu_manager.go:414] "containerMap: removing container" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369677    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369688    5237 cpu_manager.go:414] "containerMap: removing container" podUID="adf802a7-4ee5-439e-8748-8d390c5f1f51" containerName="payload-pre-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369695    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adf802a7-4ee5-439e-8748-8d390c5f1f51" containerName="payload-pre-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369708    5237 cpu_manager.go:414] "containerMap: removing container" podUID="f48a7944-1a16-42d6-8483-e6d2533ca2b1" containerName="maas-api"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369715    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f48a7944-1a16-42d6-8483-e6d2533ca2b1" containerName="maas-api"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369727    5237 cpu_manager.go:414] "containerMap: removing container" podUID="7bde0d49-ee70-4f96-991b-2f38e8984dcf" containerName="maas-api"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369735    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bde0d49-ee70-4f96-991b-2f38e8984dcf" containerName="maas-api"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369746    5237 cpu_manager.go:414] "containerMap: removing container" podUID="ac52ab6c-a555-494b-b6b9-654d2d60c8e8" containerName="maas-api"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369753    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac52ab6c-a555-494b-b6b9-654d2d60c8e8" containerName="maas-api"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369762    5237 cpu_manager.go:414] "containerMap: removing container" podUID="68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369769    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369781    5237 cpu_manager.go:414] "containerMap: removing container" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369788    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369797    5237 cpu_manager.go:414] "containerMap: removing container" podUID="f4d213de-c3e2-4455-ab48-e78d0f72b169" containerName="payload-pre-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369805    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d213de-c3e2-4455-ab48-e78d0f72b169" containerName="payload-pre-processing"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369816    5237 cpu_manager.go:414] "containerMap: removing container" podUID="2b4a25ae-6965-4713-ab67-327317f678d8" containerName="istio-proxy"
Nov 03 19:54:00.378481 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.369824    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b4a25ae-6965-4713-ab67-327317f678d8" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.401802    5237 cpu_manager.go:414] "containerMap: removing container" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.401854    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.401876    5237 cpu_manager.go:414] "containerMap: removing container" podUID="22e9f189-e173-4b40-b4a2-3455be004378" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.401884    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e9f189-e173-4b40-b4a2-3455be004378" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.401901    5237 cpu_manager.go:414] "containerMap: removing container" podUID="e6a4dcf8-18ac-443b-b78b-014a4e38ce34" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.401911    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a4dcf8-18ac-443b-b78b-014a4e38ce34" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.401928    5237 cpu_manager.go:414] "containerMap: removing container" podUID="68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.401936    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.401945    5237 cpu_manager.go:414] "containerMap: removing container" podUID="17572c14-846c-4c5d-a141-6d36045da219" containerName="maas-api"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.401952    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17572c14-846c-4c5d-a141-6d36045da219" containerName="maas-api"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.401964    5237 cpu_manager.go:414] "containerMap: removing container" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.401971    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.401983    5237 cpu_manager.go:414] "containerMap: removing container" podUID="af9b6d3c-e4c9-478c-9d19-36444a7c4e77" containerName="payload-processing"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.401991    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af9b6d3c-e4c9-478c-9d19-36444a7c4e77" containerName="payload-processing"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402115    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="af4841d0-e4fd-4544-8924-cc21f3737772" containerName="main"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402128    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4d0eae8-11ce-483c-97f7-e0aef82ed54b" containerName="payload-processing"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402145    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae76152-1c87-4685-8074-6bcc1760268c" containerName="main"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402156    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fdb8431-30cd-4023-94e9-47f32412e2e7" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402165    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e9e459-f64b-44b6-a217-56d4e11feada" containerName="payload-pre-processing"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402175    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e11f2e-c05f-4ba6-b748-328c152574a9" containerName="main"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402185    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63a1c5a-826e-487e-b3eb-eab4278b8049" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402194    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe7148f3-a2df-43bb-bf04-d5a0fddc1a13" containerName="payload-pre-processing"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402206    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82e4bbb-88bb-4b6b-b75d-d187d68e3c47" containerName="main"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402215    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a61b39a-3f50-4ff7-88fe-dc8f851db36b" containerName="main"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402223    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dacc3ed-4e8e-46d3-aa6c-7326bd160bfb" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402231    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd104015-ca9a-4193-bf07-f26376a66366" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402243    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="93bee0fc-7374-4419-ac8a-07745bdc3a24" containerName="maas-api"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402254    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e7b0c6-d3f5-43f2-bebc-0cb3c65742f2" containerName="main"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402264    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56d7264-7d35-4dd8-946b-c1ab4d40e941" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402275    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51fb02e-9d48-4c36-9911-1c077bcc915e" containerName="main"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402284    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="84914ee8-d996-409e-a212-d29119cc5de8" containerName="payload-processing"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402293    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e6fabb-7410-46bc-a67f-40d9116f02eb" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402306    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39a1945-62e0-42c7-9b2a-d81d2ee042d6" containerName="main"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402316    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="941d6f5f-47cd-439e-9463-ecc633799b05" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402326    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2acf71f-02e6-4f7f-aa4a-4daa381bc69f" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402335    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb69380-947e-474e-974f-9e17065379cf" containerName="main"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402344    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="9145fea2-1fed-40d6-82e4-2f79350f9d8d" containerName="maas-api"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402354    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1faf58-6636-4c3c-a702-53c2fe91d649" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402363    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d73f5ff-57e8-4ed0-bc2c-e00ee280a7cd" containerName="manager"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402374    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="6037ccaa-48d9-450c-8726-213ae6b76500" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402383    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d820e94-7b5a-48c6-8351-a45ea23de377" containerName="payload-pre-processing"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402392    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fbc3df5-49ef-4bbf-8a95-726f0d42f5ae" containerName="maas-api"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402402    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad53de78-95ea-4f6b-b6a2-caa3e86bd49b" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402411    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="215c21e1-690b-4a4b-ae98-277e56811956" containerName="main"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402420    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48f6c83-ae97-4927-9d94-2d17fd679536" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402429    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="adf802a7-4ee5-439e-8748-8d390c5f1f51" containerName="payload-pre-processing"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402439    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="f48a7944-1a16-42d6-8483-e6d2533ca2b1" containerName="maas-api"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402447    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bde0d49-ee70-4f96-991b-2f38e8984dcf" containerName="maas-api"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402457    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac52ab6c-a555-494b-b6b9-654d2d60c8e8" containerName="maas-api"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402466    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402476    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf93f15-df9e-4a66-8d89-4d412d3bf35c" containerName="main"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402483    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d213de-c3e2-4455-ab48-e78d0f72b169" containerName="payload-pre-processing"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402492    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b4a25ae-6965-4713-ab67-327317f678d8" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402503    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="27821d19-33d6-4db9-a259-a7a555d03e15" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402511    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e9f189-e173-4b40-b4a2-3455be004378" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402519    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a4dcf8-18ac-443b-b78b-014a4e38ce34" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402527    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="68bcd4b4-76d8-4e68-ac9c-cabeaa560b7a" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402536    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="17572c14-846c-4c5d-a141-6d36045da219" containerName="maas-api"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402544    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb974992-d3ab-4992-8ce4-d639d550163e" containerName="main"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402553    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="af9b6d3c-e4c9-478c-9d19-36444a7c4e77" containerName="payload-processing"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402562    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="0398a799-4e11-4f8e-913f-8a42574c0ae9" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402572    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="3882ca5a-6385-41e0-892c-21f22d58ddc9" containerName="payload-processing"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402579    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b14de1-629b-4f1a-b35c-cd23b30eff51" containerName="payload-processing"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402589    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="704a8596-0c3b-403f-b847-7ce2256eb13d" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402597    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fea1109-4466-401f-8de1-3c9fae530f70" containerName="istio-proxy"
Nov 03 19:54:00.403594 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.402608    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="98b3d450-df16-4c35-8095-4d0640c7a422" containerName="maas-api"
Nov 03 19:54:00.457484 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.412762    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-694f8bb88d-vrmx4"]
Nov 03 19:54:00.457484 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.412911    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694f8bb88d-vrmx4"
Nov 03 19:54:00.457484 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.425128    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 19:54:00.457484 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.425232    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 19:54:00.457484 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.425284    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wgc2d\""
Nov 03 19:54:00.457484 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.425351    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 19:54:00.584933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.562355    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca18671d-8b9a-42ef-923d-821db74fd245-metrics-certs\") pod \"maas-controller-694f8bb88d-vrmx4\" (UID: \"ca18671d-8b9a-42ef-923d-821db74fd245\") " pod="opendatahub/maas-controller-694f8bb88d-vrmx4"
Nov 03 19:54:00.584933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.562419    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ca18671d-8b9a-42ef-923d-821db74fd245-webhook-cert\") pod \"maas-controller-694f8bb88d-vrmx4\" (UID: \"ca18671d-8b9a-42ef-923d-821db74fd245\") " pod="opendatahub/maas-controller-694f8bb88d-vrmx4"
Nov 03 19:54:00.584933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.562491    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qttv\" (UniqueName: \"kubernetes.io/projected/ca18671d-8b9a-42ef-923d-821db74fd245-kube-api-access-7qttv\") pod \"maas-controller-694f8bb88d-vrmx4\" (UID: \"ca18671d-8b9a-42ef-923d-821db74fd245\") " pod="opendatahub/maas-controller-694f8bb88d-vrmx4"
Nov 03 19:54:00.656861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.611135    5237 generic.go:356] "Generic (PLEG): container finished" podID="6ff03678-8a48-4bdb-83c5-4941236f45bc" containerID="11fa0d117b7eb1bfd38ec5cb6e216ace2d7d87e5c89c67d0400546bb1ea1c78a" exitCode=0
Nov 03 19:54:00.656861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.611295    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp" event={"ID":"6ff03678-8a48-4bdb-83c5-4941236f45bc","Type":"ContainerDied","Data":"11fa0d117b7eb1bfd38ec5cb6e216ace2d7d87e5c89c67d0400546bb1ea1c78a"}
Nov 03 19:54:00.656861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.614902    5237 generic.go:356] "Generic (PLEG): container finished" podID="7ca1424b-adf4-42f6-8165-44af258b508c" containerID="863cc691deb31ce822c416bf6f606e67b0889175f0f4e6ea7c48f6039d709292" exitCode=0
Nov 03 19:54:00.656861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.615004    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9" event={"ID":"7ca1424b-adf4-42f6-8165-44af258b508c","Type":"ContainerDied","Data":"863cc691deb31ce822c416bf6f606e67b0889175f0f4e6ea7c48f6039d709292"}
Nov 03 19:54:00.656861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.639169    5237 generic.go:356] "Generic (PLEG): container finished" podID="e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" containerID="0a8ce2fb5a7ef6fbfb1dceb94852294f49490865af064c4cb1c6e59702abdaaa" exitCode=0
Nov 03 19:54:00.656861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.639228    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc" event={"ID":"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3","Type":"ContainerDied","Data":"0a8ce2fb5a7ef6fbfb1dceb94852294f49490865af064c4cb1c6e59702abdaaa"}
Nov 03 19:54:00.664136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.664102    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca18671d-8b9a-42ef-923d-821db74fd245-metrics-certs\") pod \"maas-controller-694f8bb88d-vrmx4\" (UID: \"ca18671d-8b9a-42ef-923d-821db74fd245\") " pod="opendatahub/maas-controller-694f8bb88d-vrmx4"
Nov 03 19:54:00.664357 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.664337    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ca18671d-8b9a-42ef-923d-821db74fd245-webhook-cert\") pod \"maas-controller-694f8bb88d-vrmx4\" (UID: \"ca18671d-8b9a-42ef-923d-821db74fd245\") " pod="opendatahub/maas-controller-694f8bb88d-vrmx4"
Nov 03 19:54:00.664532 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.664512    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qttv\" (UniqueName: \"kubernetes.io/projected/ca18671d-8b9a-42ef-923d-821db74fd245-kube-api-access-7qttv\") pod \"maas-controller-694f8bb88d-vrmx4\" (UID: \"ca18671d-8b9a-42ef-923d-821db74fd245\") " pod="opendatahub/maas-controller-694f8bb88d-vrmx4"
Nov 03 19:54:00.702338 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.685544    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ca18671d-8b9a-42ef-923d-821db74fd245-webhook-cert\") pod \"maas-controller-694f8bb88d-vrmx4\" (UID: \"ca18671d-8b9a-42ef-923d-821db74fd245\") " pod="opendatahub/maas-controller-694f8bb88d-vrmx4"
Nov 03 19:54:00.714179 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.711401    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca18671d-8b9a-42ef-923d-821db74fd245-metrics-certs\") pod \"maas-controller-694f8bb88d-vrmx4\" (UID: \"ca18671d-8b9a-42ef-923d-821db74fd245\") " pod="opendatahub/maas-controller-694f8bb88d-vrmx4"
Nov 03 19:54:00.759906 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.750358    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:54:00.773785 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.773751    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qttv\" (UniqueName: \"kubernetes.io/projected/ca18671d-8b9a-42ef-923d-821db74fd245-kube-api-access-7qttv\") pod \"maas-controller-694f8bb88d-vrmx4\" (UID: \"ca18671d-8b9a-42ef-923d-821db74fd245\") " pod="opendatahub/maas-controller-694f8bb88d-vrmx4"
Nov 03 19:54:00.839962 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.824064    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:54:00.889269 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.889216    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-workload-socket\") pod \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") "
Nov 03 19:54:00.889574 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.889532    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-podinfo\") pod \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") "
Nov 03 19:54:00.889805 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.889737    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istiod-ca-cert\") pod \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") "
Nov 03 19:54:00.890010 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.889788    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-envoy\") pod \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") "
Nov 03 19:54:00.890242 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.890207    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-data\") pod \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") "
Nov 03 19:54:00.905331 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.890374    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-kube-api-access-gj9xx\" (UniqueName: \"kubernetes.io/projected/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-kube-api-access-gj9xx\") pod \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") "
Nov 03 19:54:00.905331 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.892931    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-workload-certs\") pod \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") "
Nov 03 19:54:00.905331 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.893012    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-token\" (UniqueName: \"kubernetes.io/projected/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-token\") pod \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") "
Nov 03 19:54:00.905331 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.893139    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-credential-socket\") pod \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\" (UID: \"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3\") "
Nov 03 19:54:00.905331 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.903025    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istiod-ca-cert" pod "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" (UID: "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:54:00.905331 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.904912    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-workload-socket" pod "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" (UID: "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:54:00.906014 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.905405    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:00.906014 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.905428    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:00.948973 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.924950    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-credential-socket" pod "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" (UID: "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:54:00.948973 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.925734    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-data" pod "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" (UID: "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:54:00.948973 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.934338    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-podinfo" pod "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" (UID: "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:54:00.948973 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.944953    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-workload-certs" pod "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" (UID: "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:54:00.950474 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.950125    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-token" pod "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" (UID: "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:54:00.950653 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.950516    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-kube-api-access-gj9xx" pod "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" (UID: "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3"). InnerVolumeSpecName "kube-api-access-gj9xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:54:00.951126 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.950910    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-envoy" pod "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" (UID: "e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:54:00.961024 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:00.961000    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:54:01.008859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.008103    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-workload-socket\") pod \"7ca1424b-adf4-42f6-8165-44af258b508c\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") "
Nov 03 19:54:01.008859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.008222    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca1424b-adf4-42f6-8165-44af258b508c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca1424b-adf4-42f6-8165-44af258b508c-istiod-ca-cert\") pod \"7ca1424b-adf4-42f6-8165-44af258b508c\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") "
Nov 03 19:54:01.008859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.008265    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-workload-certs\") pod \"7ca1424b-adf4-42f6-8165-44af258b508c\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") "
Nov 03 19:54:01.008859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.008316    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca1424b-adf4-42f6-8165-44af258b508c-istio-token\" (UniqueName: \"kubernetes.io/projected/7ca1424b-adf4-42f6-8165-44af258b508c-istio-token\") pod \"7ca1424b-adf4-42f6-8165-44af258b508c\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") "
Nov 03 19:54:01.008859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.008378    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-istio-data\") pod \"7ca1424b-adf4-42f6-8165-44af258b508c\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") "
Nov 03 19:54:01.008859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.008421    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-credential-socket\") pod \"7ca1424b-adf4-42f6-8165-44af258b508c\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") "
Nov 03 19:54:01.008859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.008457    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ca1424b-adf4-42f6-8165-44af258b508c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca1424b-adf4-42f6-8165-44af258b508c-istio-podinfo\") pod \"7ca1424b-adf4-42f6-8165-44af258b508c\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") "
Nov 03 19:54:01.008859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.008490    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca1424b-adf4-42f6-8165-44af258b508c-kube-api-access-84vtw\" (UniqueName: \"kubernetes.io/projected/7ca1424b-adf4-42f6-8165-44af258b508c-kube-api-access-84vtw\") pod \"7ca1424b-adf4-42f6-8165-44af258b508c\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") "
Nov 03 19:54:01.008859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.008523    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-istio-envoy\") pod \"7ca1424b-adf4-42f6-8165-44af258b508c\" (UID: \"7ca1424b-adf4-42f6-8165-44af258b508c\") "
Nov 03 19:54:01.009536 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.008928    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.009536 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.008953    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.009536 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.008969    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.009536 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.008982    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.009536 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.008999    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj9xx\" (UniqueName: \"kubernetes.io/projected/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-kube-api-access-gj9xx\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.009536 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.009015    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.009536 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.009029    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.040859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.021950    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-workload-socket" pod "7ca1424b-adf4-42f6-8165-44af258b508c" (UID: "7ca1424b-adf4-42f6-8165-44af258b508c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:54:01.040859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.022396    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca1424b-adf4-42f6-8165-44af258b508c-istiod-ca-cert" pod "7ca1424b-adf4-42f6-8165-44af258b508c" (UID: "7ca1424b-adf4-42f6-8165-44af258b508c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:54:01.040859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.027929    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-workload-certs" pod "7ca1424b-adf4-42f6-8165-44af258b508c" (UID: "7ca1424b-adf4-42f6-8165-44af258b508c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:54:01.040859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.040140    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-istio-data" pod "7ca1424b-adf4-42f6-8165-44af258b508c" (UID: "7ca1424b-adf4-42f6-8165-44af258b508c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:54:01.040859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.040351    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-credential-socket" pod "7ca1424b-adf4-42f6-8165-44af258b508c" (UID: "7ca1424b-adf4-42f6-8165-44af258b508c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:54:01.048859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.043575    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca1424b-adf4-42f6-8165-44af258b508c-istio-token" pod "7ca1424b-adf4-42f6-8165-44af258b508c" (UID: "7ca1424b-adf4-42f6-8165-44af258b508c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:54:01.048859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.043772    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-istio-envoy" pod "7ca1424b-adf4-42f6-8165-44af258b508c" (UID: "7ca1424b-adf4-42f6-8165-44af258b508c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:54:01.048859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.044162    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ca1424b-adf4-42f6-8165-44af258b508c-istio-podinfo" pod "7ca1424b-adf4-42f6-8165-44af258b508c" (UID: "7ca1424b-adf4-42f6-8165-44af258b508c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:54:01.048859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.045969    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca1424b-adf4-42f6-8165-44af258b508c-kube-api-access-84vtw" pod "7ca1424b-adf4-42f6-8165-44af258b508c" (UID: "7ca1424b-adf4-42f6-8165-44af258b508c"). InnerVolumeSpecName "kube-api-access-84vtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:54:01.057128 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.056977    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694f8bb88d-vrmx4"
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111116    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-workload-certs\") pod \"6ff03678-8a48-4bdb-83c5-4941236f45bc\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") "
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111180    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff03678-8a48-4bdb-83c5-4941236f45bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff03678-8a48-4bdb-83c5-4941236f45bc-istiod-ca-cert\") pod \"6ff03678-8a48-4bdb-83c5-4941236f45bc\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") "
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111238    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-data\") pod \"6ff03678-8a48-4bdb-83c5-4941236f45bc\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") "
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111326    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff03678-8a48-4bdb-83c5-4941236f45bc-kube-api-access-n625j\" (UniqueName: \"kubernetes.io/projected/6ff03678-8a48-4bdb-83c5-4941236f45bc-kube-api-access-n625j\") pod \"6ff03678-8a48-4bdb-83c5-4941236f45bc\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") "
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111370    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-token\" (UniqueName: \"kubernetes.io/projected/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-token\") pod \"6ff03678-8a48-4bdb-83c5-4941236f45bc\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") "
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111410    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-workload-socket\") pod \"6ff03678-8a48-4bdb-83c5-4941236f45bc\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") "
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111456    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-envoy\") pod \"6ff03678-8a48-4bdb-83c5-4941236f45bc\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") "
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111498    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-podinfo\") pod \"6ff03678-8a48-4bdb-83c5-4941236f45bc\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") "
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111535    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-credential-socket\") pod \"6ff03678-8a48-4bdb-83c5-4941236f45bc\" (UID: \"6ff03678-8a48-4bdb-83c5-4941236f45bc\") "
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111808    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111848    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca1424b-adf4-42f6-8165-44af258b508c-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111864    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84vtw\" (UniqueName: \"kubernetes.io/projected/7ca1424b-adf4-42f6-8165-44af258b508c-kube-api-access-84vtw\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111881    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111895    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111912    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca1424b-adf4-42f6-8165-44af258b508c-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111925    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111938    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca1424b-adf4-42f6-8165-44af258b508c-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.119309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.111950    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca1424b-adf4-42f6-8165-44af258b508c-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.145992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.128193    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff03678-8a48-4bdb-83c5-4941236f45bc-istiod-ca-cert" pod "6ff03678-8a48-4bdb-83c5-4941236f45bc" (UID: "6ff03678-8a48-4bdb-83c5-4941236f45bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:54:01.145992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.128512    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-workload-certs" pod "6ff03678-8a48-4bdb-83c5-4941236f45bc" (UID: "6ff03678-8a48-4bdb-83c5-4941236f45bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:54:01.145992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.134915    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-data" pod "6ff03678-8a48-4bdb-83c5-4941236f45bc" (UID: "6ff03678-8a48-4bdb-83c5-4941236f45bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:54:01.145992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.145487    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-credential-socket" pod "6ff03678-8a48-4bdb-83c5-4941236f45bc" (UID: "6ff03678-8a48-4bdb-83c5-4941236f45bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:54:01.145992 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.145884    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-workload-socket" pod "6ff03678-8a48-4bdb-83c5-4941236f45bc" (UID: "6ff03678-8a48-4bdb-83c5-4941236f45bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:54:01.148690 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.148423    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff03678-8a48-4bdb-83c5-4941236f45bc-kube-api-access-n625j" pod "6ff03678-8a48-4bdb-83c5-4941236f45bc" (UID: "6ff03678-8a48-4bdb-83c5-4941236f45bc"). InnerVolumeSpecName "kube-api-access-n625j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:54:01.148690 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.148508    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-envoy" pod "6ff03678-8a48-4bdb-83c5-4941236f45bc" (UID: "6ff03678-8a48-4bdb-83c5-4941236f45bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:54:01.148690 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.148569    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-token" pod "6ff03678-8a48-4bdb-83c5-4941236f45bc" (UID: "6ff03678-8a48-4bdb-83c5-4941236f45bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:54:01.148690 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.148624    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-podinfo" pod "6ff03678-8a48-4bdb-83c5-4941236f45bc" (UID: "6ff03678-8a48-4bdb-83c5-4941236f45bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:54:01.224887 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.224554    5237 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-workload-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.224887 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.224586    5237 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-envoy\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.224887 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.224601    5237 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-podinfo\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.224887 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.224616    5237 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-credential-socket\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.224887 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.224630    5237 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-workload-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.224887 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.224644    5237 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff03678-8a48-4bdb-83c5-4941236f45bc-istiod-ca-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.224887 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.224658    5237 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-data\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.224887 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.224672    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n625j\" (UniqueName: \"kubernetes.io/projected/6ff03678-8a48-4bdb-83c5-4941236f45bc-kube-api-access-n625j\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.224887 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.224687    5237 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff03678-8a48-4bdb-83c5-4941236f45bc-istio-token\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:01.439791 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.439406    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-694f8bb88d-vrmx4"]
Nov 03 19:54:01.447942 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.447782    5237 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:54:01.648322 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.648281    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"
Nov 03 19:54:01.648810 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.648287    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc" event={"ID":"e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3","Type":"ContainerDied","Data":"e07feca7b3aaaf69aa68a38c2a3e169fdfa6456c1d391fabecd975f11122a8a4"}
Nov 03 19:54:01.648810 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.648439    5237 scope.go:122] "RemoveContainer" containerID="0a8ce2fb5a7ef6fbfb1dceb94852294f49490865af064c4cb1c6e59702abdaaa"
Nov 03 19:54:01.649830 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.649810    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"
Nov 03 19:54:01.649948 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.649830    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp" event={"ID":"6ff03678-8a48-4bdb-83c5-4941236f45bc","Type":"ContainerDied","Data":"e1b94f36c08b876818e614a42543109cd41bb633b4095345abd057ee0e65e486"}
Nov 03 19:54:01.651136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.651099    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694f8bb88d-vrmx4" event={"ID":"ca18671d-8b9a-42ef-923d-821db74fd245","Type":"ContainerStarted","Data":"1ae320d2eeffaf8e6fc0750e29c0c0d74aae9498c23ed8a9ae1d9cf00aaa37a6"}
Nov 03 19:54:01.652901 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.652879    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9" event={"ID":"7ca1424b-adf4-42f6-8165-44af258b508c","Type":"ContainerDied","Data":"1af3400e649de1402548e2accdeca79e7d2af0b23a370961bd52635da1df3f54"}
Nov 03 19:54:01.652993 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.652907    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"
Nov 03 19:54:01.659683 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.659663    5237 scope.go:122] "RemoveContainer" containerID="11fa0d117b7eb1bfd38ec5cb6e216ace2d7d87e5c89c67d0400546bb1ea1c78a"
Nov 03 19:54:01.701063 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.701031    5237 scope.go:122] "RemoveContainer" containerID="863cc691deb31ce822c416bf6f606e67b0889175f0f4e6ea7c48f6039d709292"
Nov 03 19:54:01.719957 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.712570    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"]
Nov 03 19:54:01.740860 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.722654    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-zzdfc"]
Nov 03 19:54:01.748914 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.748866    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"]
Nov 03 19:54:01.751034 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.751009    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-n7mpp"]
Nov 03 19:54:01.763997 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.763973    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"]
Nov 03 19:54:01.774864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.769003    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-lhzz9"]
Nov 03 19:54:01.865376 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.865339    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff03678-8a48-4bdb-83c5-4941236f45bc" path="/var/lib/kubelet/pods/6ff03678-8a48-4bdb-83c5-4941236f45bc/volumes"
Nov 03 19:54:01.865761 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.865746    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca1424b-adf4-42f6-8165-44af258b508c" path="/var/lib/kubelet/pods/7ca1424b-adf4-42f6-8165-44af258b508c/volumes"
Nov 03 19:54:01.866170 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:01.866155    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" path="/var/lib/kubelet/pods/e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3/volumes"
Nov 03 19:54:02.663505 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:02.663457    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694f8bb88d-vrmx4" event={"ID":"ca18671d-8b9a-42ef-923d-821db74fd245","Type":"ContainerStarted","Data":"ae9f690185ab619f665a9a8eab4b4fc6606184420eb34eee1c67b73e22cf2794"}
Nov 03 19:54:02.663937 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:02.663571    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-694f8bb88d-vrmx4"
Nov 03 19:54:02.685107 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:02.685044    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-694f8bb88d-vrmx4" podStartSLOduration=2.179181234 podStartE2EDuration="2.685029239s" totalImagesPullingTime="505.848005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:02.683613575 +0000 UTC m=+1598.937920823" watchObservedRunningTime="2026-11-03 19:54:02.685029239 +0000 UTC m=+1598.939336463"
Nov 03 19:54:04.022743 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:04.022710    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:04.179136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:04.179096    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:05.116467 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:05.116430    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:10.742187 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:10.742147    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:12.461525 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:12.461489    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:12.595311 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:12.595276    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:13.255253 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:54:13.255216    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:54:13.671451 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:13.671418    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-694f8bb88d-vrmx4"
Nov 03 19:54:19.418861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.418812    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"]
Nov 03 19:54:19.419375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.419193    5237 cpu_manager.go:414] "containerMap: removing container" podUID="7ca1424b-adf4-42f6-8165-44af258b508c" containerName="istio-proxy"
Nov 03 19:54:19.419375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.419210    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca1424b-adf4-42f6-8165-44af258b508c" containerName="istio-proxy"
Nov 03 19:54:19.419375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.419228    5237 cpu_manager.go:414] "containerMap: removing container" podUID="6ff03678-8a48-4bdb-83c5-4941236f45bc" containerName="istio-proxy"
Nov 03 19:54:19.419375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.419236    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff03678-8a48-4bdb-83c5-4941236f45bc" containerName="istio-proxy"
Nov 03 19:54:19.419375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.419258    5237 cpu_manager.go:414] "containerMap: removing container" podUID="e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" containerName="istio-proxy"
Nov 03 19:54:19.419375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.419263    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" containerName="istio-proxy"
Nov 03 19:54:19.419375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.419323    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c97a36-e042-4c1d-8d68-3b08f7e6c3d3" containerName="istio-proxy"
Nov 03 19:54:19.419375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.419335    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca1424b-adf4-42f6-8165-44af258b508c" containerName="istio-proxy"
Nov 03 19:54:19.419375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.419340    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff03678-8a48-4bdb-83c5-4941236f45bc" containerName="istio-proxy"
Nov 03 19:54:19.421877 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.421855    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"
Nov 03 19:54:19.425903 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.425880    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:54:19.427270 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.427255    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\""
Nov 03 19:54:19.431686 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.431664    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"]
Nov 03 19:54:19.505584 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.505550    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fed683d-6d7c-4a17-895a-c3689b7ec02d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-q9r7z\" (UID: \"5fed683d-6d7c-4a17-895a-c3689b7ec02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"
Nov 03 19:54:19.505761 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.505617    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67zm9\" (UniqueName: \"kubernetes.io/projected/5fed683d-6d7c-4a17-895a-c3689b7ec02d-kube-api-access-67zm9\") pod \"maas-api-revoke-sub-85237e45-q9r7z\" (UID: \"5fed683d-6d7c-4a17-895a-c3689b7ec02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"
Nov 03 19:54:19.505761 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.505715    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fed683d-6d7c-4a17-895a-c3689b7ec02d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-q9r7z\" (UID: \"5fed683d-6d7c-4a17-895a-c3689b7ec02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"
Nov 03 19:54:19.508899 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.508876    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:54:19.508975 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.508953    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:54:19.606827 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.606790    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fed683d-6d7c-4a17-895a-c3689b7ec02d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-q9r7z\" (UID: \"5fed683d-6d7c-4a17-895a-c3689b7ec02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"
Nov 03 19:54:19.607024 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.606881    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zm9\" (UniqueName: \"kubernetes.io/projected/5fed683d-6d7c-4a17-895a-c3689b7ec02d-kube-api-access-67zm9\") pod \"maas-api-revoke-sub-85237e45-q9r7z\" (UID: \"5fed683d-6d7c-4a17-895a-c3689b7ec02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"
Nov 03 19:54:19.607024 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.606914    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fed683d-6d7c-4a17-895a-c3689b7ec02d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-q9r7z\" (UID: \"5fed683d-6d7c-4a17-895a-c3689b7ec02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"
Nov 03 19:54:19.607582 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.607556    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fed683d-6d7c-4a17-895a-c3689b7ec02d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-q9r7z\" (UID: \"5fed683d-6d7c-4a17-895a-c3689b7ec02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"
Nov 03 19:54:19.609286 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.609261    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fed683d-6d7c-4a17-895a-c3689b7ec02d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-q9r7z\" (UID: \"5fed683d-6d7c-4a17-895a-c3689b7ec02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"
Nov 03 19:54:19.619614 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.619590    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zm9\" (UniqueName: \"kubernetes.io/projected/5fed683d-6d7c-4a17-895a-c3689b7ec02d-kube-api-access-67zm9\") pod \"maas-api-revoke-sub-85237e45-q9r7z\" (UID: \"5fed683d-6d7c-4a17-895a-c3689b7ec02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"
Nov 03 19:54:19.734880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:19.734762    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"
Nov 03 19:54:20.009375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:20.009298    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"]
Nov 03 19:54:20.012554 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:54:20.012520    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fed683d_6d7c_4a17_895a_c3689b7ec02d.slice/crio-73a1284346b96a41351b9d81f4905562e5bfa1cb96cb0b4835a2dd8ed026f89c WatchSource:0}: Error finding container 73a1284346b96a41351b9d81f4905562e5bfa1cb96cb0b4835a2dd8ed026f89c: Status 404 returned error can't find the container with id 73a1284346b96a41351b9d81f4905562e5bfa1cb96cb0b4835a2dd8ed026f89c
Nov 03 19:54:20.744063 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:20.744027    5237 generic.go:356] "Generic (PLEG): container finished" podID="5fed683d-6d7c-4a17-895a-c3689b7ec02d" containerID="ba41c00da425fa52a1f962fd12f2a9d52d14314992c9667eb3cf9de0c771045c" exitCode=0
Nov 03 19:54:20.744441 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:20.744089    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z" event={"ID":"5fed683d-6d7c-4a17-895a-c3689b7ec02d","Type":"ContainerDied","Data":"ba41c00da425fa52a1f962fd12f2a9d52d14314992c9667eb3cf9de0c771045c"}
Nov 03 19:54:20.744441 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:20.744130    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z" event={"ID":"5fed683d-6d7c-4a17-895a-c3689b7ec02d","Type":"ContainerStarted","Data":"73a1284346b96a41351b9d81f4905562e5bfa1cb96cb0b4835a2dd8ed026f89c"}
Nov 03 19:54:21.879066 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:21.879040    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"
Nov 03 19:54:21.929773 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:21.929741    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fed683d-6d7c-4a17-895a-c3689b7ec02d-kube-api-access-67zm9\" (UniqueName: \"kubernetes.io/projected/5fed683d-6d7c-4a17-895a-c3689b7ec02d-kube-api-access-67zm9\") pod \"5fed683d-6d7c-4a17-895a-c3689b7ec02d\" (UID: \"5fed683d-6d7c-4a17-895a-c3689b7ec02d\") "
Nov 03 19:54:21.929773 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:21.929780    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fed683d-6d7c-4a17-895a-c3689b7ec02d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fed683d-6d7c-4a17-895a-c3689b7ec02d-maas-api-openshift-ca\") pod \"5fed683d-6d7c-4a17-895a-c3689b7ec02d\" (UID: \"5fed683d-6d7c-4a17-895a-c3689b7ec02d\") "
Nov 03 19:54:21.930019 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:21.929811    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fed683d-6d7c-4a17-895a-c3689b7ec02d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fed683d-6d7c-4a17-895a-c3689b7ec02d-maas-api-xks-ca\") pod \"5fed683d-6d7c-4a17-895a-c3689b7ec02d\" (UID: \"5fed683d-6d7c-4a17-895a-c3689b7ec02d\") "
Nov 03 19:54:21.930251 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:21.930225    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fed683d-6d7c-4a17-895a-c3689b7ec02d-maas-api-openshift-ca" pod "5fed683d-6d7c-4a17-895a-c3689b7ec02d" (UID: "5fed683d-6d7c-4a17-895a-c3689b7ec02d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:54:21.932094 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:21.932068    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fed683d-6d7c-4a17-895a-c3689b7ec02d-kube-api-access-67zm9" pod "5fed683d-6d7c-4a17-895a-c3689b7ec02d" (UID: "5fed683d-6d7c-4a17-895a-c3689b7ec02d"). InnerVolumeSpecName "kube-api-access-67zm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:54:21.932094 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:21.932067    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fed683d-6d7c-4a17-895a-c3689b7ec02d-maas-api-xks-ca" pod "5fed683d-6d7c-4a17-895a-c3689b7ec02d" (UID: "5fed683d-6d7c-4a17-895a-c3689b7ec02d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:54:22.031470 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:22.031388    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67zm9\" (UniqueName: \"kubernetes.io/projected/5fed683d-6d7c-4a17-895a-c3689b7ec02d-kube-api-access-67zm9\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:22.031470 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:22.031418    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fed683d-6d7c-4a17-895a-c3689b7ec02d-maas-api-openshift-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:22.031470 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:22.031428    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fed683d-6d7c-4a17-895a-c3689b7ec02d-maas-api-xks-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:22.753849 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:22.753804    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"
Nov 03 19:54:22.754030 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:22.753867    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z" event={"ID":"5fed683d-6d7c-4a17-895a-c3689b7ec02d","Type":"ContainerDied","Data":"73a1284346b96a41351b9d81f4905562e5bfa1cb96cb0b4835a2dd8ed026f89c"}
Nov 03 19:54:22.754030 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:22.753920    5237 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73a1284346b96a41351b9d81f4905562e5bfa1cb96cb0b4835a2dd8ed026f89c"
Nov 03 19:54:24.592026 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:24.591992    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:24.738533 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:24.738496    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:30.786337 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:30.786295    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:31.178402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.178294    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:31.311185 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.311143    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:31.365874 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.365815    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-694f8bb88d-vrmx4"]
Nov 03 19:54:31.366086 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.366058    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-694f8bb88d-vrmx4" podUID="ca18671d-8b9a-42ef-923d-821db74fd245" containerName="manager" containerID="cri-o://ae9f690185ab619f665a9a8eab4b4fc6606184420eb34eee1c67b73e22cf2794" gracePeriod=30
Nov 03 19:54:31.602654 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.602630    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694f8bb88d-vrmx4"
Nov 03 19:54:31.719897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.719865    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca18671d-8b9a-42ef-923d-821db74fd245-webhook-cert\" (UniqueName: \"kubernetes.io/secret/ca18671d-8b9a-42ef-923d-821db74fd245-webhook-cert\") pod \"ca18671d-8b9a-42ef-923d-821db74fd245\" (UID: \"ca18671d-8b9a-42ef-923d-821db74fd245\") "
Nov 03 19:54:31.720052 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.719903    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca18671d-8b9a-42ef-923d-821db74fd245-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca18671d-8b9a-42ef-923d-821db74fd245-metrics-certs\") pod \"ca18671d-8b9a-42ef-923d-821db74fd245\" (UID: \"ca18671d-8b9a-42ef-923d-821db74fd245\") "
Nov 03 19:54:31.720052 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.719966    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca18671d-8b9a-42ef-923d-821db74fd245-kube-api-access-7qttv\" (UniqueName: \"kubernetes.io/projected/ca18671d-8b9a-42ef-923d-821db74fd245-kube-api-access-7qttv\") pod \"ca18671d-8b9a-42ef-923d-821db74fd245\" (UID: \"ca18671d-8b9a-42ef-923d-821db74fd245\") "
Nov 03 19:54:31.722146 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.722112    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca18671d-8b9a-42ef-923d-821db74fd245-metrics-certs" pod "ca18671d-8b9a-42ef-923d-821db74fd245" (UID: "ca18671d-8b9a-42ef-923d-821db74fd245"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:54:31.722276 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.722154    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca18671d-8b9a-42ef-923d-821db74fd245-webhook-cert" pod "ca18671d-8b9a-42ef-923d-821db74fd245" (UID: "ca18671d-8b9a-42ef-923d-821db74fd245"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:54:31.722276 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.722159    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca18671d-8b9a-42ef-923d-821db74fd245-kube-api-access-7qttv" pod "ca18671d-8b9a-42ef-923d-821db74fd245" (UID: "ca18671d-8b9a-42ef-923d-821db74fd245"). InnerVolumeSpecName "kube-api-access-7qttv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:54:31.793946 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.793908    5237 generic.go:356] "Generic (PLEG): container finished" podID="ca18671d-8b9a-42ef-923d-821db74fd245" containerID="ae9f690185ab619f665a9a8eab4b4fc6606184420eb34eee1c67b73e22cf2794" exitCode=0
Nov 03 19:54:31.794377 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.793964    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694f8bb88d-vrmx4"
Nov 03 19:54:31.794377 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.793989    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694f8bb88d-vrmx4" event={"ID":"ca18671d-8b9a-42ef-923d-821db74fd245","Type":"ContainerDied","Data":"ae9f690185ab619f665a9a8eab4b4fc6606184420eb34eee1c67b73e22cf2794"}
Nov 03 19:54:31.794377 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.794045    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694f8bb88d-vrmx4" event={"ID":"ca18671d-8b9a-42ef-923d-821db74fd245","Type":"ContainerDied","Data":"1ae320d2eeffaf8e6fc0750e29c0c0d74aae9498c23ed8a9ae1d9cf00aaa37a6"}
Nov 03 19:54:31.794377 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.794067    5237 scope.go:122] "RemoveContainer" containerID="ae9f690185ab619f665a9a8eab4b4fc6606184420eb34eee1c67b73e22cf2794"
Nov 03 19:54:31.802938 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.802907    5237 scope.go:122] "RemoveContainer" containerID="ae9f690185ab619f665a9a8eab4b4fc6606184420eb34eee1c67b73e22cf2794"
Nov 03 19:54:31.803191 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:54:31.803171    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9f690185ab619f665a9a8eab4b4fc6606184420eb34eee1c67b73e22cf2794\": container with ID starting with ae9f690185ab619f665a9a8eab4b4fc6606184420eb34eee1c67b73e22cf2794 not found: ID does not exist" containerID="ae9f690185ab619f665a9a8eab4b4fc6606184420eb34eee1c67b73e22cf2794"
Nov 03 19:54:31.803241 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.803201    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9f690185ab619f665a9a8eab4b4fc6606184420eb34eee1c67b73e22cf2794"} err="failed to get container status \"ae9f690185ab619f665a9a8eab4b4fc6606184420eb34eee1c67b73e22cf2794\": rpc error: code = NotFound desc = could not find container \"ae9f690185ab619f665a9a8eab4b4fc6606184420eb34eee1c67b73e22cf2794\": container with ID starting with ae9f690185ab619f665a9a8eab4b4fc6606184420eb34eee1c67b73e22cf2794 not found: ID does not exist"
Nov 03 19:54:31.815374 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.815349    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-694f8bb88d-vrmx4"]
Nov 03 19:54:31.818905 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.818882    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-694f8bb88d-vrmx4"]
Nov 03 19:54:31.821041 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.821021    5237 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ca18671d-8b9a-42ef-923d-821db74fd245-webhook-cert\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:31.821123 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.821044    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca18671d-8b9a-42ef-923d-821db74fd245-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:31.821123 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.821062    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qttv\" (UniqueName: \"kubernetes.io/projected/ca18671d-8b9a-42ef-923d-821db74fd245-kube-api-access-7qttv\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:31.858297 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:31.858257    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca18671d-8b9a-42ef-923d-821db74fd245" path="/var/lib/kubelet/pods/ca18671d-8b9a-42ef-923d-821db74fd245/volumes"
Nov 03 19:54:33.064732 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.064692    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:33.272184 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.272152    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-694f8bb88d-h7fff"]
Nov 03 19:54:33.272491 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.272480    5237 cpu_manager.go:414] "containerMap: removing container" podUID="5fed683d-6d7c-4a17-895a-c3689b7ec02d" containerName="revoke-keys"
Nov 03 19:54:33.272535 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.272492    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fed683d-6d7c-4a17-895a-c3689b7ec02d" containerName="revoke-keys"
Nov 03 19:54:33.272535 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.272508    5237 cpu_manager.go:414] "containerMap: removing container" podUID="ca18671d-8b9a-42ef-923d-821db74fd245" containerName="manager"
Nov 03 19:54:33.272535 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.272513    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca18671d-8b9a-42ef-923d-821db74fd245" containerName="manager"
Nov 03 19:54:33.272626 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.272571    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca18671d-8b9a-42ef-923d-821db74fd245" containerName="manager"
Nov 03 19:54:33.272626 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.272585    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fed683d-6d7c-4a17-895a-c3689b7ec02d" containerName="revoke-keys"
Nov 03 19:54:33.276681 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.276664    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694f8bb88d-h7fff"
Nov 03 19:54:33.280916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.280896    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 19:54:33.287073 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.287047    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-694f8bb88d-h7fff"]
Nov 03 19:54:33.322578 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.322501    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 19:54:33.322578 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.322502    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wgc2d\""
Nov 03 19:54:33.322578 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.322502    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 19:54:33.333963 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.333937    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z99mr\" (UniqueName: \"kubernetes.io/projected/9a72f77e-2af9-4e75-bbfe-a20447358ef5-kube-api-access-z99mr\") pod \"maas-controller-694f8bb88d-h7fff\" (UID: \"9a72f77e-2af9-4e75-bbfe-a20447358ef5\") " pod="opendatahub/maas-controller-694f8bb88d-h7fff"
Nov 03 19:54:33.334087 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.334067    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a72f77e-2af9-4e75-bbfe-a20447358ef5-metrics-certs\") pod \"maas-controller-694f8bb88d-h7fff\" (UID: \"9a72f77e-2af9-4e75-bbfe-a20447358ef5\") " pod="opendatahub/maas-controller-694f8bb88d-h7fff"
Nov 03 19:54:33.334155 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.334120    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a72f77e-2af9-4e75-bbfe-a20447358ef5-webhook-cert\") pod \"maas-controller-694f8bb88d-h7fff\" (UID: \"9a72f77e-2af9-4e75-bbfe-a20447358ef5\") " pod="opendatahub/maas-controller-694f8bb88d-h7fff"
Nov 03 19:54:33.434997 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.434951    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a72f77e-2af9-4e75-bbfe-a20447358ef5-metrics-certs\") pod \"maas-controller-694f8bb88d-h7fff\" (UID: \"9a72f77e-2af9-4e75-bbfe-a20447358ef5\") " pod="opendatahub/maas-controller-694f8bb88d-h7fff"
Nov 03 19:54:33.435207 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.435009    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a72f77e-2af9-4e75-bbfe-a20447358ef5-webhook-cert\") pod \"maas-controller-694f8bb88d-h7fff\" (UID: \"9a72f77e-2af9-4e75-bbfe-a20447358ef5\") " pod="opendatahub/maas-controller-694f8bb88d-h7fff"
Nov 03 19:54:33.435207 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.435056    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z99mr\" (UniqueName: \"kubernetes.io/projected/9a72f77e-2af9-4e75-bbfe-a20447358ef5-kube-api-access-z99mr\") pod \"maas-controller-694f8bb88d-h7fff\" (UID: \"9a72f77e-2af9-4e75-bbfe-a20447358ef5\") " pod="opendatahub/maas-controller-694f8bb88d-h7fff"
Nov 03 19:54:33.437525 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.437497    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a72f77e-2af9-4e75-bbfe-a20447358ef5-webhook-cert\") pod \"maas-controller-694f8bb88d-h7fff\" (UID: \"9a72f77e-2af9-4e75-bbfe-a20447358ef5\") " pod="opendatahub/maas-controller-694f8bb88d-h7fff"
Nov 03 19:54:33.437525 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.437515    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a72f77e-2af9-4e75-bbfe-a20447358ef5-metrics-certs\") pod \"maas-controller-694f8bb88d-h7fff\" (UID: \"9a72f77e-2af9-4e75-bbfe-a20447358ef5\") " pod="opendatahub/maas-controller-694f8bb88d-h7fff"
Nov 03 19:54:33.447734 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.447712    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z99mr\" (UniqueName: \"kubernetes.io/projected/9a72f77e-2af9-4e75-bbfe-a20447358ef5-kube-api-access-z99mr\") pod \"maas-controller-694f8bb88d-h7fff\" (UID: \"9a72f77e-2af9-4e75-bbfe-a20447358ef5\") " pod="opendatahub/maas-controller-694f8bb88d-h7fff"
Nov 03 19:54:33.590753 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.590655    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694f8bb88d-h7fff"
Nov 03 19:54:33.868816 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:33.868740    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-694f8bb88d-h7fff"]
Nov 03 19:54:33.872084 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:54:33.872056    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a72f77e_2af9_4e75_bbfe_a20447358ef5.slice/crio-2f6411b9d0ad87bf9e49b37448833dc62b410896fed210bcd050d78ce7038aa0 WatchSource:0}: Error finding container 2f6411b9d0ad87bf9e49b37448833dc62b410896fed210bcd050d78ce7038aa0: Status 404 returned error can't find the container with id 2f6411b9d0ad87bf9e49b37448833dc62b410896fed210bcd050d78ce7038aa0
Nov 03 19:54:34.809556 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:34.809514    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694f8bb88d-h7fff" event={"ID":"9a72f77e-2af9-4e75-bbfe-a20447358ef5","Type":"ContainerStarted","Data":"edfc79db3f8af384cc073a9fb97319ebffce654d4c3c84c735b92115622f5893"}
Nov 03 19:54:34.809963 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:34.809578    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694f8bb88d-h7fff" event={"ID":"9a72f77e-2af9-4e75-bbfe-a20447358ef5","Type":"ContainerStarted","Data":"2f6411b9d0ad87bf9e49b37448833dc62b410896fed210bcd050d78ce7038aa0"}
Nov 03 19:54:34.809963 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:34.809696    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-694f8bb88d-h7fff"
Nov 03 19:54:34.832714 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:34.832661    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-694f8bb88d-h7fff" podStartSLOduration=1.356434257 podStartE2EDuration="1.832646204s" totalImagesPullingTime="476.211947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:34.831664411 +0000 UTC m=+1631.085971633" watchObservedRunningTime="2026-11-03 19:54:34.832646204 +0000 UTC m=+1631.086953428"
Nov 03 19:54:43.254527 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:54:43.254489    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:54:45.816627 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:54:45.816593    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-694f8bb88d-h7fff"
Nov 03 19:55:11.231006 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:11.230958    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-f5xfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:55:11.231809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:11.231775    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb" podUID="98ec8c1c-49db-492e-958e-fabe09d044fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:55:13.256282 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:55:13.256236    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:55:15.930801 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:15.930770    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"]
Nov 03 19:55:15.933977 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:15.933958    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"
Nov 03 19:55:15.939734 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:15.939709    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:55:15.939734 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:15.939726    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:55:15.940016 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:15.939729    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:55:15.940016 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:15.939805    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\""
Nov 03 19:55:15.943919 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:15.943893    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"]
Nov 03 19:55:16.056726 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:16.056691    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghcm8\" (UniqueName: \"kubernetes.io/projected/59cbdbc9-2495-4374-b013-12777a2bc983-kube-api-access-ghcm8\") pod \"maas-api-revoke-sub-c1e271a9-vr2ph\" (UID: \"59cbdbc9-2495-4374-b013-12777a2bc983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"
Nov 03 19:55:16.056726 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:16.056727    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59cbdbc9-2495-4374-b013-12777a2bc983-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vr2ph\" (UID: \"59cbdbc9-2495-4374-b013-12777a2bc983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"
Nov 03 19:55:16.057025 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:16.056749    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59cbdbc9-2495-4374-b013-12777a2bc983-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vr2ph\" (UID: \"59cbdbc9-2495-4374-b013-12777a2bc983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"
Nov 03 19:55:16.157638 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:16.157601    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghcm8\" (UniqueName: \"kubernetes.io/projected/59cbdbc9-2495-4374-b013-12777a2bc983-kube-api-access-ghcm8\") pod \"maas-api-revoke-sub-c1e271a9-vr2ph\" (UID: \"59cbdbc9-2495-4374-b013-12777a2bc983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"
Nov 03 19:55:16.157638 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:16.157638    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59cbdbc9-2495-4374-b013-12777a2bc983-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vr2ph\" (UID: \"59cbdbc9-2495-4374-b013-12777a2bc983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"
Nov 03 19:55:16.157883 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:16.157661    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59cbdbc9-2495-4374-b013-12777a2bc983-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vr2ph\" (UID: \"59cbdbc9-2495-4374-b013-12777a2bc983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"
Nov 03 19:55:16.158318 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:16.158290    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59cbdbc9-2495-4374-b013-12777a2bc983-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vr2ph\" (UID: \"59cbdbc9-2495-4374-b013-12777a2bc983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"
Nov 03 19:55:16.160359 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:16.160335    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59cbdbc9-2495-4374-b013-12777a2bc983-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vr2ph\" (UID: \"59cbdbc9-2495-4374-b013-12777a2bc983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"
Nov 03 19:55:16.169888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:16.169862    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghcm8\" (UniqueName: \"kubernetes.io/projected/59cbdbc9-2495-4374-b013-12777a2bc983-kube-api-access-ghcm8\") pod \"maas-api-revoke-sub-c1e271a9-vr2ph\" (UID: \"59cbdbc9-2495-4374-b013-12777a2bc983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"
Nov 03 19:55:16.248903 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:16.248792    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"
Nov 03 19:55:16.523796 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:16.523705    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"]
Nov 03 19:55:16.526969 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:55:16.526929    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59cbdbc9_2495_4374_b013_12777a2bc983.slice/crio-1d1a397a4a06b44755349b504d604fa4f9140d2322afddc9a878576ee2b08e34 WatchSource:0}: Error finding container 1d1a397a4a06b44755349b504d604fa4f9140d2322afddc9a878576ee2b08e34: Status 404 returned error can't find the container with id 1d1a397a4a06b44755349b504d604fa4f9140d2322afddc9a878576ee2b08e34
Nov 03 19:55:16.994934 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:16.994895    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph" event={"ID":"59cbdbc9-2495-4374-b013-12777a2bc983","Type":"ContainerStarted","Data":"750133c4bf10bfe68380bb8473af910c3d9c22f9aad18cc7d1c5f108d88f4376"}
Nov 03 19:55:16.995375 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:16.994956    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph" event={"ID":"59cbdbc9-2495-4374-b013-12777a2bc983","Type":"ContainerStarted","Data":"1d1a397a4a06b44755349b504d604fa4f9140d2322afddc9a878576ee2b08e34"}
Nov 03 19:55:17.015180 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:17.015126    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph" podStartSLOduration=2.015110341 podStartE2EDuration="2.015110341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:17.013811535 +0000 UTC m=+1673.268118760" watchObservedRunningTime="2026-11-03 19:55:17.015110341 +0000 UTC m=+1673.269417606"
Nov 03 19:55:18.000938 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:18.000900    5237 generic.go:356] "Generic (PLEG): container finished" podID="59cbdbc9-2495-4374-b013-12777a2bc983" containerID="750133c4bf10bfe68380bb8473af910c3d9c22f9aad18cc7d1c5f108d88f4376" exitCode=0
Nov 03 19:55:18.001401 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:18.000965    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph" event={"ID":"59cbdbc9-2495-4374-b013-12777a2bc983","Type":"ContainerDied","Data":"750133c4bf10bfe68380bb8473af910c3d9c22f9aad18cc7d1c5f108d88f4376"}
Nov 03 19:55:19.133551 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:19.133521    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"
Nov 03 19:55:19.287865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:19.287815    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59cbdbc9-2495-4374-b013-12777a2bc983-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59cbdbc9-2495-4374-b013-12777a2bc983-maas-api-xks-ca\") pod \"59cbdbc9-2495-4374-b013-12777a2bc983\" (UID: \"59cbdbc9-2495-4374-b013-12777a2bc983\") "
Nov 03 19:55:19.288057 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:19.287975    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59cbdbc9-2495-4374-b013-12777a2bc983-kube-api-access-ghcm8\" (UniqueName: \"kubernetes.io/projected/59cbdbc9-2495-4374-b013-12777a2bc983-kube-api-access-ghcm8\") pod \"59cbdbc9-2495-4374-b013-12777a2bc983\" (UID: \"59cbdbc9-2495-4374-b013-12777a2bc983\") "
Nov 03 19:55:19.288057 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:19.288014    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59cbdbc9-2495-4374-b013-12777a2bc983-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59cbdbc9-2495-4374-b013-12777a2bc983-maas-api-openshift-ca\") pod \"59cbdbc9-2495-4374-b013-12777a2bc983\" (UID: \"59cbdbc9-2495-4374-b013-12777a2bc983\") "
Nov 03 19:55:19.288391 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:19.288365    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59cbdbc9-2495-4374-b013-12777a2bc983-maas-api-openshift-ca" pod "59cbdbc9-2495-4374-b013-12777a2bc983" (UID: "59cbdbc9-2495-4374-b013-12777a2bc983"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:19.289985 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:19.289958    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59cbdbc9-2495-4374-b013-12777a2bc983-maas-api-xks-ca" pod "59cbdbc9-2495-4374-b013-12777a2bc983" (UID: "59cbdbc9-2495-4374-b013-12777a2bc983"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:55:19.290091 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:19.290010    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59cbdbc9-2495-4374-b013-12777a2bc983-kube-api-access-ghcm8" pod "59cbdbc9-2495-4374-b013-12777a2bc983" (UID: "59cbdbc9-2495-4374-b013-12777a2bc983"). InnerVolumeSpecName "kube-api-access-ghcm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:19.389647 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:19.389593    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghcm8\" (UniqueName: \"kubernetes.io/projected/59cbdbc9-2495-4374-b013-12777a2bc983-kube-api-access-ghcm8\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:19.389647 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:19.389639    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59cbdbc9-2495-4374-b013-12777a2bc983-maas-api-openshift-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:19.389647 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:19.389652    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59cbdbc9-2495-4374-b013-12777a2bc983-maas-api-xks-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:20.011187 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:20.011154    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"
Nov 03 19:55:20.011187 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:20.011158    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph" event={"ID":"59cbdbc9-2495-4374-b013-12777a2bc983","Type":"ContainerDied","Data":"1d1a397a4a06b44755349b504d604fa4f9140d2322afddc9a878576ee2b08e34"}
Nov 03 19:55:20.011402 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:20.011213    5237 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d1a397a4a06b44755349b504d604fa4f9140d2322afddc9a878576ee2b08e34"
Nov 03 19:55:30.912012 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:30.911967    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:31.380862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:31.378659    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:34.822637 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.822601    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"]
Nov 03 19:55:34.823064 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.822954    5237 cpu_manager.go:414] "containerMap: removing container" podUID="59cbdbc9-2495-4374-b013-12777a2bc983" containerName="revoke-keys"
Nov 03 19:55:34.823064 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.822967    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59cbdbc9-2495-4374-b013-12777a2bc983" containerName="revoke-keys"
Nov 03 19:55:34.823064 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.823039    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="59cbdbc9-2495-4374-b013-12777a2bc983" containerName="revoke-keys"
Nov 03 19:55:34.827406 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.827389    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"
Nov 03 19:55:34.848263 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.848235    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"]
Nov 03 19:55:34.852201 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.852172    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8fds\" (UniqueName: \"kubernetes.io/projected/c6b00742-24ad-45a6-a535-b0100043b95c-kube-api-access-h8fds\") pod \"maas-api-revoke-sub-2e1f5b5f-bw8wj\" (UID: \"c6b00742-24ad-45a6-a535-b0100043b95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"
Nov 03 19:55:34.853172 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.852237    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b00742-24ad-45a6-a535-b0100043b95c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bw8wj\" (UID: \"c6b00742-24ad-45a6-a535-b0100043b95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"
Nov 03 19:55:34.853172 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.852363    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b00742-24ad-45a6-a535-b0100043b95c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bw8wj\" (UID: \"c6b00742-24ad-45a6-a535-b0100043b95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"
Nov 03 19:55:34.853172 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.852925    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:55:34.854199 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.854179    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:55:34.854317 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.854217    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:55:34.854317 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.854221    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\""
Nov 03 19:55:34.953587 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.953547    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8fds\" (UniqueName: \"kubernetes.io/projected/c6b00742-24ad-45a6-a535-b0100043b95c-kube-api-access-h8fds\") pod \"maas-api-revoke-sub-2e1f5b5f-bw8wj\" (UID: \"c6b00742-24ad-45a6-a535-b0100043b95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"
Nov 03 19:55:34.953804 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.953634    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b00742-24ad-45a6-a535-b0100043b95c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bw8wj\" (UID: \"c6b00742-24ad-45a6-a535-b0100043b95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"
Nov 03 19:55:34.953804 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.953689    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b00742-24ad-45a6-a535-b0100043b95c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bw8wj\" (UID: \"c6b00742-24ad-45a6-a535-b0100043b95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"
Nov 03 19:55:34.954463 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.954442    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b00742-24ad-45a6-a535-b0100043b95c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bw8wj\" (UID: \"c6b00742-24ad-45a6-a535-b0100043b95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"
Nov 03 19:55:34.956116 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.956094    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b00742-24ad-45a6-a535-b0100043b95c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bw8wj\" (UID: \"c6b00742-24ad-45a6-a535-b0100043b95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"
Nov 03 19:55:34.968965 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:34.968933    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8fds\" (UniqueName: \"kubernetes.io/projected/c6b00742-24ad-45a6-a535-b0100043b95c-kube-api-access-h8fds\") pod \"maas-api-revoke-sub-2e1f5b5f-bw8wj\" (UID: \"c6b00742-24ad-45a6-a535-b0100043b95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"
Nov 03 19:55:35.139754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:35.139655    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"
Nov 03 19:55:35.416649 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:35.416575    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"]
Nov 03 19:55:35.419809 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:55:35.419780    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b00742_24ad_45a6_a535_b0100043b95c.slice/crio-4a746eec3d22ca9e5502c91336e9fc561669bb460094dcc31b8fa91624bcfd3d WatchSource:0}: Error finding container 4a746eec3d22ca9e5502c91336e9fc561669bb460094dcc31b8fa91624bcfd3d: Status 404 returned error can't find the container with id 4a746eec3d22ca9e5502c91336e9fc561669bb460094dcc31b8fa91624bcfd3d
Nov 03 19:55:36.084676 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:36.084637    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj" event={"ID":"c6b00742-24ad-45a6-a535-b0100043b95c","Type":"ContainerStarted","Data":"47e91deed890138e907804a3752790b8779a3e4158b0cc8a165e6cb87e976439"}
Nov 03 19:55:36.085206 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:36.084697    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj" event={"ID":"c6b00742-24ad-45a6-a535-b0100043b95c","Type":"ContainerStarted","Data":"4a746eec3d22ca9e5502c91336e9fc561669bb460094dcc31b8fa91624bcfd3d"}
Nov 03 19:55:36.105153 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:36.105098    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj" podStartSLOduration=2.105083927 podStartE2EDuration="2.105083927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:36.103388999 +0000 UTC m=+1692.357696223" watchObservedRunningTime="2026-11-03 19:55:36.105083927 +0000 UTC m=+1692.359391151"
Nov 03 19:55:37.092098 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:37.092055    5237 generic.go:356] "Generic (PLEG): container finished" podID="c6b00742-24ad-45a6-a535-b0100043b95c" containerID="47e91deed890138e907804a3752790b8779a3e4158b0cc8a165e6cb87e976439" exitCode=0
Nov 03 19:55:37.092542 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:37.092142    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj" event={"ID":"c6b00742-24ad-45a6-a535-b0100043b95c","Type":"ContainerDied","Data":"47e91deed890138e907804a3752790b8779a3e4158b0cc8a165e6cb87e976439"}
Nov 03 19:55:38.221629 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:38.221600    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"
Nov 03 19:55:38.285416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:38.285386    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6b00742-24ad-45a6-a535-b0100043b95c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b00742-24ad-45a6-a535-b0100043b95c-maas-api-openshift-ca\") pod \"c6b00742-24ad-45a6-a535-b0100043b95c\" (UID: \"c6b00742-24ad-45a6-a535-b0100043b95c\") "
Nov 03 19:55:38.285624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:38.285492    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6b00742-24ad-45a6-a535-b0100043b95c-kube-api-access-h8fds\" (UniqueName: \"kubernetes.io/projected/c6b00742-24ad-45a6-a535-b0100043b95c-kube-api-access-h8fds\") pod \"c6b00742-24ad-45a6-a535-b0100043b95c\" (UID: \"c6b00742-24ad-45a6-a535-b0100043b95c\") "
Nov 03 19:55:38.285624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:38.285558    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6b00742-24ad-45a6-a535-b0100043b95c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b00742-24ad-45a6-a535-b0100043b95c-maas-api-xks-ca\") pod \"c6b00742-24ad-45a6-a535-b0100043b95c\" (UID: \"c6b00742-24ad-45a6-a535-b0100043b95c\") "
Nov 03 19:55:38.285830 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:38.285807    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b00742-24ad-45a6-a535-b0100043b95c-maas-api-openshift-ca" pod "c6b00742-24ad-45a6-a535-b0100043b95c" (UID: "c6b00742-24ad-45a6-a535-b0100043b95c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:38.287635 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:38.287602    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6b00742-24ad-45a6-a535-b0100043b95c-maas-api-xks-ca" pod "c6b00742-24ad-45a6-a535-b0100043b95c" (UID: "c6b00742-24ad-45a6-a535-b0100043b95c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:55:38.287745 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:38.287644    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b00742-24ad-45a6-a535-b0100043b95c-kube-api-access-h8fds" pod "c6b00742-24ad-45a6-a535-b0100043b95c" (UID: "c6b00742-24ad-45a6-a535-b0100043b95c"). InnerVolumeSpecName "kube-api-access-h8fds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:38.387206 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:38.387122    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b00742-24ad-45a6-a535-b0100043b95c-maas-api-openshift-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:38.387206 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:38.387155    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8fds\" (UniqueName: \"kubernetes.io/projected/c6b00742-24ad-45a6-a535-b0100043b95c-kube-api-access-h8fds\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:38.387206 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:38.387166    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b00742-24ad-45a6-a535-b0100043b95c-maas-api-xks-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:39.103081 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:39.103049    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"
Nov 03 19:55:39.103081 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:39.103056    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj" event={"ID":"c6b00742-24ad-45a6-a535-b0100043b95c","Type":"ContainerDied","Data":"4a746eec3d22ca9e5502c91336e9fc561669bb460094dcc31b8fa91624bcfd3d"}
Nov 03 19:55:39.103334 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:39.103106    5237 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a746eec3d22ca9e5502c91336e9fc561669bb460094dcc31b8fa91624bcfd3d"
Nov 03 19:55:43.255549 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:55:43.255507    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:55:45.825194 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:45.825154    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"]
Nov 03 19:55:45.825579 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:45.825520    5237 cpu_manager.go:414] "containerMap: removing container" podUID="c6b00742-24ad-45a6-a535-b0100043b95c" containerName="revoke-keys"
Nov 03 19:55:45.825579 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:45.825532    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b00742-24ad-45a6-a535-b0100043b95c" containerName="revoke-keys"
Nov 03 19:55:45.825650 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:45.825606    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b00742-24ad-45a6-a535-b0100043b95c" containerName="revoke-keys"
Nov 03 19:55:45.828565 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:45.828547    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"
Nov 03 19:55:45.837102 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:45.837073    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:55:45.837256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:45.837170    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\""
Nov 03 19:55:45.839624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:45.839601    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:55:45.839756 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:45.839735    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:55:45.842656 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:45.842616    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"]
Nov 03 19:55:45.956959 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:45.956920    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f87mt\" (UniqueName: \"kubernetes.io/projected/da6f60e0-ab2b-4c15-8030-995c1a7a243d-kube-api-access-f87mt\") pod \"maas-api-revoke-sub-ec843018-dhnfj\" (UID: \"da6f60e0-ab2b-4c15-8030-995c1a7a243d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"
Nov 03 19:55:45.957145 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:45.956986    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6f60e0-ab2b-4c15-8030-995c1a7a243d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-dhnfj\" (UID: \"da6f60e0-ab2b-4c15-8030-995c1a7a243d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"
Nov 03 19:55:45.957145 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:45.957094    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6f60e0-ab2b-4c15-8030-995c1a7a243d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-dhnfj\" (UID: \"da6f60e0-ab2b-4c15-8030-995c1a7a243d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"
Nov 03 19:55:46.058202 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:46.058158    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87mt\" (UniqueName: \"kubernetes.io/projected/da6f60e0-ab2b-4c15-8030-995c1a7a243d-kube-api-access-f87mt\") pod \"maas-api-revoke-sub-ec843018-dhnfj\" (UID: \"da6f60e0-ab2b-4c15-8030-995c1a7a243d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"
Nov 03 19:55:46.058404 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:46.058213    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6f60e0-ab2b-4c15-8030-995c1a7a243d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-dhnfj\" (UID: \"da6f60e0-ab2b-4c15-8030-995c1a7a243d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"
Nov 03 19:55:46.058404 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:46.058246    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6f60e0-ab2b-4c15-8030-995c1a7a243d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-dhnfj\" (UID: \"da6f60e0-ab2b-4c15-8030-995c1a7a243d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"
Nov 03 19:55:46.058969 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:46.058948    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6f60e0-ab2b-4c15-8030-995c1a7a243d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-dhnfj\" (UID: \"da6f60e0-ab2b-4c15-8030-995c1a7a243d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"
Nov 03 19:55:46.060640 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:46.060618    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6f60e0-ab2b-4c15-8030-995c1a7a243d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-dhnfj\" (UID: \"da6f60e0-ab2b-4c15-8030-995c1a7a243d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"
Nov 03 19:55:46.071415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:46.071392    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f87mt\" (UniqueName: \"kubernetes.io/projected/da6f60e0-ab2b-4c15-8030-995c1a7a243d-kube-api-access-f87mt\") pod \"maas-api-revoke-sub-ec843018-dhnfj\" (UID: \"da6f60e0-ab2b-4c15-8030-995c1a7a243d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"
Nov 03 19:55:46.145476 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:46.145390    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"
Nov 03 19:55:46.427435 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:46.427311    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"]
Nov 03 19:55:46.430146 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:55:46.430115    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6f60e0_ab2b_4c15_8030_995c1a7a243d.slice/crio-3e180d36110a411dc2103d54fd59926c5440b2881aa2859f9aaa55ffe54e1bdc WatchSource:0}: Error finding container 3e180d36110a411dc2103d54fd59926c5440b2881aa2859f9aaa55ffe54e1bdc: Status 404 returned error can't find the container with id 3e180d36110a411dc2103d54fd59926c5440b2881aa2859f9aaa55ffe54e1bdc
Nov 03 19:55:47.145028 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:47.144996    5237 generic.go:356] "Generic (PLEG): container finished" podID="da6f60e0-ab2b-4c15-8030-995c1a7a243d" containerID="c5c8f06a4a3151d37e7fcdebec17725322faa60e1ec98fd30daed300da243d07" exitCode=0
Nov 03 19:55:47.145434 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:47.145093    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj" event={"ID":"da6f60e0-ab2b-4c15-8030-995c1a7a243d","Type":"ContainerDied","Data":"c5c8f06a4a3151d37e7fcdebec17725322faa60e1ec98fd30daed300da243d07"}
Nov 03 19:55:47.145434 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:47.145147    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj" event={"ID":"da6f60e0-ab2b-4c15-8030-995c1a7a243d","Type":"ContainerStarted","Data":"3e180d36110a411dc2103d54fd59926c5440b2881aa2859f9aaa55ffe54e1bdc"}
Nov 03 19:55:47.704240 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:47.704208    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:47.849590 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:47.849548    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:48.319137 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:48.319102    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"
Nov 03 19:55:48.379327 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:48.379286    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da6f60e0-ab2b-4c15-8030-995c1a7a243d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6f60e0-ab2b-4c15-8030-995c1a7a243d-maas-api-xks-ca\") pod \"da6f60e0-ab2b-4c15-8030-995c1a7a243d\" (UID: \"da6f60e0-ab2b-4c15-8030-995c1a7a243d\") "
Nov 03 19:55:48.379540 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:48.379346    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da6f60e0-ab2b-4c15-8030-995c1a7a243d-kube-api-access-f87mt\" (UniqueName: \"kubernetes.io/projected/da6f60e0-ab2b-4c15-8030-995c1a7a243d-kube-api-access-f87mt\") pod \"da6f60e0-ab2b-4c15-8030-995c1a7a243d\" (UID: \"da6f60e0-ab2b-4c15-8030-995c1a7a243d\") "
Nov 03 19:55:48.379540 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:48.379443    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da6f60e0-ab2b-4c15-8030-995c1a7a243d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6f60e0-ab2b-4c15-8030-995c1a7a243d-maas-api-openshift-ca\") pod \"da6f60e0-ab2b-4c15-8030-995c1a7a243d\" (UID: \"da6f60e0-ab2b-4c15-8030-995c1a7a243d\") "
Nov 03 19:55:48.380084 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:48.380054    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6f60e0-ab2b-4c15-8030-995c1a7a243d-maas-api-openshift-ca" pod "da6f60e0-ab2b-4c15-8030-995c1a7a243d" (UID: "da6f60e0-ab2b-4c15-8030-995c1a7a243d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:48.382013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:48.381990    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6f60e0-ab2b-4c15-8030-995c1a7a243d-kube-api-access-f87mt" pod "da6f60e0-ab2b-4c15-8030-995c1a7a243d" (UID: "da6f60e0-ab2b-4c15-8030-995c1a7a243d"). InnerVolumeSpecName "kube-api-access-f87mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:48.382013 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:48.381996    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da6f60e0-ab2b-4c15-8030-995c1a7a243d-maas-api-xks-ca" pod "da6f60e0-ab2b-4c15-8030-995c1a7a243d" (UID: "da6f60e0-ab2b-4c15-8030-995c1a7a243d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:55:48.481316 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:48.481224    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6f60e0-ab2b-4c15-8030-995c1a7a243d-maas-api-xks-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:48.481316 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:48.481269    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f87mt\" (UniqueName: \"kubernetes.io/projected/da6f60e0-ab2b-4c15-8030-995c1a7a243d-kube-api-access-f87mt\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:48.481316 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:48.481286    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6f60e0-ab2b-4c15-8030-995c1a7a243d-maas-api-openshift-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:49.154649 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:49.154622    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"
Nov 03 19:55:49.154846 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:49.154655    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj" event={"ID":"da6f60e0-ab2b-4c15-8030-995c1a7a243d","Type":"ContainerDied","Data":"3e180d36110a411dc2103d54fd59926c5440b2881aa2859f9aaa55ffe54e1bdc"}
Nov 03 19:55:49.154846 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:49.154709    5237 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e180d36110a411dc2103d54fd59926c5440b2881aa2859f9aaa55ffe54e1bdc"
Nov 03 19:55:49.258642 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:55:49.258604    5237 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6f60e0_ab2b_4c15_8030_995c1a7a243d.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 19:55:50.363034 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:50.362986    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-t9ft8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:55:50.363484 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:50.363054    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8" podUID="c8d65667-e957-4334-a43a-71d03f1d1000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:55:50.415030 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:50.414969    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-bqzwl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:55:50.415371 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:55:50.415321    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl" podUID="f5da6be6-a4c0-40f5-8f07-d3de1d188394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:12.585405 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.585362    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"]
Nov 03 19:56:12.585981 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.585929    5237 cpu_manager.go:414] "containerMap: removing container" podUID="da6f60e0-ab2b-4c15-8030-995c1a7a243d" containerName="revoke-keys"
Nov 03 19:56:12.585981 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.585956    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da6f60e0-ab2b-4c15-8030-995c1a7a243d" containerName="revoke-keys"
Nov 03 19:56:12.586118 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.586066    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="da6f60e0-ab2b-4c15-8030-995c1a7a243d" containerName="revoke-keys"
Nov 03 19:56:12.589004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.588980    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"
Nov 03 19:56:12.594245 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.594209    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:56:12.594388 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.594352    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:56:12.594388 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.594352    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:56:12.594388 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.594356    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\""
Nov 03 19:56:12.598189 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.598164    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"]
Nov 03 19:56:12.647161 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.647129    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6336a7a-38a0-4d29-855e-192cf3b737f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-bg6ps\" (UID: \"f6336a7a-38a0-4d29-855e-192cf3b737f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"
Nov 03 19:56:12.647349 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.647173    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvkmq\" (UniqueName: \"kubernetes.io/projected/f6336a7a-38a0-4d29-855e-192cf3b737f7-kube-api-access-zvkmq\") pod \"maas-api-revoke-sub-046e1343-bg6ps\" (UID: \"f6336a7a-38a0-4d29-855e-192cf3b737f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"
Nov 03 19:56:12.647349 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.647216    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6336a7a-38a0-4d29-855e-192cf3b737f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-bg6ps\" (UID: \"f6336a7a-38a0-4d29-855e-192cf3b737f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"
Nov 03 19:56:12.748455 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.748413    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6336a7a-38a0-4d29-855e-192cf3b737f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-bg6ps\" (UID: \"f6336a7a-38a0-4d29-855e-192cf3b737f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"
Nov 03 19:56:12.748640 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.748484    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6336a7a-38a0-4d29-855e-192cf3b737f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-bg6ps\" (UID: \"f6336a7a-38a0-4d29-855e-192cf3b737f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"
Nov 03 19:56:12.748640 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.748511    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvkmq\" (UniqueName: \"kubernetes.io/projected/f6336a7a-38a0-4d29-855e-192cf3b737f7-kube-api-access-zvkmq\") pod \"maas-api-revoke-sub-046e1343-bg6ps\" (UID: \"f6336a7a-38a0-4d29-855e-192cf3b737f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"
Nov 03 19:56:12.749076 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.749056    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6336a7a-38a0-4d29-855e-192cf3b737f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-bg6ps\" (UID: \"f6336a7a-38a0-4d29-855e-192cf3b737f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"
Nov 03 19:56:12.750856 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.750807    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6336a7a-38a0-4d29-855e-192cf3b737f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-bg6ps\" (UID: \"f6336a7a-38a0-4d29-855e-192cf3b737f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"
Nov 03 19:56:12.760445 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.760420    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvkmq\" (UniqueName: \"kubernetes.io/projected/f6336a7a-38a0-4d29-855e-192cf3b737f7-kube-api-access-zvkmq\") pod \"maas-api-revoke-sub-046e1343-bg6ps\" (UID: \"f6336a7a-38a0-4d29-855e-192cf3b737f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"
Nov 03 19:56:12.905256 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:12.905169    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"
Nov 03 19:56:13.176892 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:13.176800    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"]
Nov 03 19:56:13.180156 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:56:13.180119    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6336a7a_38a0_4d29_855e_192cf3b737f7.slice/crio-2fc81d613e9682da8e8465e3d14de3ba0c88f4e84e6ae24248765441910725a1 WatchSource:0}: Error finding container 2fc81d613e9682da8e8465e3d14de3ba0c88f4e84e6ae24248765441910725a1: Status 404 returned error can't find the container with id 2fc81d613e9682da8e8465e3d14de3ba0c88f4e84e6ae24248765441910725a1
Nov 03 19:56:13.254510 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:56:13.254474    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:56:13.283660 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:13.283630    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps" event={"ID":"f6336a7a-38a0-4d29-855e-192cf3b737f7","Type":"ContainerStarted","Data":"2fc81d613e9682da8e8465e3d14de3ba0c88f4e84e6ae24248765441910725a1"}
Nov 03 19:56:14.289792 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:14.289757    5237 generic.go:356] "Generic (PLEG): container finished" podID="f6336a7a-38a0-4d29-855e-192cf3b737f7" containerID="7911cb08ae1bb96239f214b349e30f23867f7dce35bc61bc8dc8bf0e3010ec6b" exitCode=0
Nov 03 19:56:14.290229 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:14.289868    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps" event={"ID":"f6336a7a-38a0-4d29-855e-192cf3b737f7","Type":"ContainerDied","Data":"7911cb08ae1bb96239f214b349e30f23867f7dce35bc61bc8dc8bf0e3010ec6b"}
Nov 03 19:56:15.421685 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:15.421661    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"
Nov 03 19:56:15.475637 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:15.475609    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6336a7a-38a0-4d29-855e-192cf3b737f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6336a7a-38a0-4d29-855e-192cf3b737f7-maas-api-xks-ca\") pod \"f6336a7a-38a0-4d29-855e-192cf3b737f7\" (UID: \"f6336a7a-38a0-4d29-855e-192cf3b737f7\") "
Nov 03 19:56:15.475828 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:15.475663    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6336a7a-38a0-4d29-855e-192cf3b737f7-kube-api-access-zvkmq\" (UniqueName: \"kubernetes.io/projected/f6336a7a-38a0-4d29-855e-192cf3b737f7-kube-api-access-zvkmq\") pod \"f6336a7a-38a0-4d29-855e-192cf3b737f7\" (UID: \"f6336a7a-38a0-4d29-855e-192cf3b737f7\") "
Nov 03 19:56:15.475828 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:15.475733    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6336a7a-38a0-4d29-855e-192cf3b737f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6336a7a-38a0-4d29-855e-192cf3b737f7-maas-api-openshift-ca\") pod \"f6336a7a-38a0-4d29-855e-192cf3b737f7\" (UID: \"f6336a7a-38a0-4d29-855e-192cf3b737f7\") "
Nov 03 19:56:15.476140 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:15.476119    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6336a7a-38a0-4d29-855e-192cf3b737f7-maas-api-openshift-ca" pod "f6336a7a-38a0-4d29-855e-192cf3b737f7" (UID: "f6336a7a-38a0-4d29-855e-192cf3b737f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:56:15.477809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:15.477785    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6336a7a-38a0-4d29-855e-192cf3b737f7-maas-api-xks-ca" pod "f6336a7a-38a0-4d29-855e-192cf3b737f7" (UID: "f6336a7a-38a0-4d29-855e-192cf3b737f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:56:15.477941 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:15.477787    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6336a7a-38a0-4d29-855e-192cf3b737f7-kube-api-access-zvkmq" pod "f6336a7a-38a0-4d29-855e-192cf3b737f7" (UID: "f6336a7a-38a0-4d29-855e-192cf3b737f7"). InnerVolumeSpecName "kube-api-access-zvkmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:15.576671 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:15.576581    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvkmq\" (UniqueName: \"kubernetes.io/projected/f6336a7a-38a0-4d29-855e-192cf3b737f7-kube-api-access-zvkmq\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:15.576671 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:15.576612    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6336a7a-38a0-4d29-855e-192cf3b737f7-maas-api-openshift-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:15.576671 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:15.576625    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6336a7a-38a0-4d29-855e-192cf3b737f7-maas-api-xks-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:16.299085 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:16.299058    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"
Nov 03 19:56:16.299262 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:16.299085    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps" event={"ID":"f6336a7a-38a0-4d29-855e-192cf3b737f7","Type":"ContainerDied","Data":"2fc81d613e9682da8e8465e3d14de3ba0c88f4e84e6ae24248765441910725a1"}
Nov 03 19:56:16.299262 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:16.299132    5237 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fc81d613e9682da8e8465e3d14de3ba0c88f4e84e6ae24248765441910725a1"
Nov 03 19:56:22.863133 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:22.863095    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"]
Nov 03 19:56:22.863516 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:22.863466    5237 cpu_manager.go:414] "containerMap: removing container" podUID="f6336a7a-38a0-4d29-855e-192cf3b737f7" containerName="revoke-keys"
Nov 03 19:56:22.863516 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:22.863478    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6336a7a-38a0-4d29-855e-192cf3b737f7" containerName="revoke-keys"
Nov 03 19:56:22.863583 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:22.863570    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6336a7a-38a0-4d29-855e-192cf3b737f7" containerName="revoke-keys"
Nov 03 19:56:22.866849 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:22.866822    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"
Nov 03 19:56:22.872072 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:22.872044    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:56:22.873809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:22.873794    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:56:22.877610 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:22.877581    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"]
Nov 03 19:56:22.887945 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:22.887916    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:56:22.888096 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:22.887994    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\""
Nov 03 19:56:22.944354 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:22.944319    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-vs6ds\" (UID: \"2b31c84e-3dbd-4dfd-a807-9f9019fa464c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"
Nov 03 19:56:22.944524 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:22.944387    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqfg7\" (UniqueName: \"kubernetes.io/projected/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-kube-api-access-kqfg7\") pod \"maas-api-revoke-sub-d3fba056-vs6ds\" (UID: \"2b31c84e-3dbd-4dfd-a807-9f9019fa464c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"
Nov 03 19:56:22.944569 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:22.944515    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-vs6ds\" (UID: \"2b31c84e-3dbd-4dfd-a807-9f9019fa464c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"
Nov 03 19:56:23.045366 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:23.045330    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-vs6ds\" (UID: \"2b31c84e-3dbd-4dfd-a807-9f9019fa464c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"
Nov 03 19:56:23.045542 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:23.045397    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-vs6ds\" (UID: \"2b31c84e-3dbd-4dfd-a807-9f9019fa464c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"
Nov 03 19:56:23.045542 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:23.045431    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqfg7\" (UniqueName: \"kubernetes.io/projected/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-kube-api-access-kqfg7\") pod \"maas-api-revoke-sub-d3fba056-vs6ds\" (UID: \"2b31c84e-3dbd-4dfd-a807-9f9019fa464c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"
Nov 03 19:56:23.046047 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:23.046025    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-vs6ds\" (UID: \"2b31c84e-3dbd-4dfd-a807-9f9019fa464c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"
Nov 03 19:56:23.047682 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:23.047657    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-vs6ds\" (UID: \"2b31c84e-3dbd-4dfd-a807-9f9019fa464c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"
Nov 03 19:56:23.058090 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:23.058066    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqfg7\" (UniqueName: \"kubernetes.io/projected/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-kube-api-access-kqfg7\") pod \"maas-api-revoke-sub-d3fba056-vs6ds\" (UID: \"2b31c84e-3dbd-4dfd-a807-9f9019fa464c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"
Nov 03 19:56:23.179735 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:23.179650    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"
Nov 03 19:56:23.452343 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:23.452316    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"]
Nov 03 19:56:23.455138 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:56:23.455112    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b31c84e_3dbd_4dfd_a807_9f9019fa464c.slice/crio-0d495de8589758d428edc48e68c775a270d3187c8dec9616c6627b6afe6ee83f WatchSource:0}: Error finding container 0d495de8589758d428edc48e68c775a270d3187c8dec9616c6627b6afe6ee83f: Status 404 returned error can't find the container with id 0d495de8589758d428edc48e68c775a270d3187c8dec9616c6627b6afe6ee83f
Nov 03 19:56:24.338342 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:24.338297    5237 generic.go:356] "Generic (PLEG): container finished" podID="2b31c84e-3dbd-4dfd-a807-9f9019fa464c" containerID="0d74c43821f6f44ce7596b4f9b6f8a69a9e0a30d12d636aefbfc6a2e2eb4ab51" exitCode=0
Nov 03 19:56:24.338897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:24.338375    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds" event={"ID":"2b31c84e-3dbd-4dfd-a807-9f9019fa464c","Type":"ContainerDied","Data":"0d74c43821f6f44ce7596b4f9b6f8a69a9e0a30d12d636aefbfc6a2e2eb4ab51"}
Nov 03 19:56:24.338897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:24.338492    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds" event={"ID":"2b31c84e-3dbd-4dfd-a807-9f9019fa464c","Type":"ContainerStarted","Data":"0d495de8589758d428edc48e68c775a270d3187c8dec9616c6627b6afe6ee83f"}
Nov 03 19:56:25.222599 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:25.222564    5237 scope.go:122] "RemoveContainer" containerID="1c9d739eb58e78a7ec8d8f9431ba30a1beea932f7ffb1068a924029b1c58c6ff"
Nov 03 19:56:25.705861 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:25.705168    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"
Nov 03 19:56:25.739927 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:25.739879    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-kube-api-access-kqfg7\" (UniqueName: \"kubernetes.io/projected/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-kube-api-access-kqfg7\") pod \"2b31c84e-3dbd-4dfd-a807-9f9019fa464c\" (UID: \"2b31c84e-3dbd-4dfd-a807-9f9019fa464c\") "
Nov 03 19:56:25.740770 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:25.740747    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-maas-api-openshift-ca\") pod \"2b31c84e-3dbd-4dfd-a807-9f9019fa464c\" (UID: \"2b31c84e-3dbd-4dfd-a807-9f9019fa464c\") "
Nov 03 19:56:25.741429 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:25.741402    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-maas-api-xks-ca\") pod \"2b31c84e-3dbd-4dfd-a807-9f9019fa464c\" (UID: \"2b31c84e-3dbd-4dfd-a807-9f9019fa464c\") "
Nov 03 19:56:25.741994 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:25.741335    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-maas-api-openshift-ca" pod "2b31c84e-3dbd-4dfd-a807-9f9019fa464c" (UID: "2b31c84e-3dbd-4dfd-a807-9f9019fa464c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:56:25.743261 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:25.743223    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-maas-api-openshift-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:25.747291 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:25.747268    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-kube-api-access-kqfg7" pod "2b31c84e-3dbd-4dfd-a807-9f9019fa464c" (UID: "2b31c84e-3dbd-4dfd-a807-9f9019fa464c"). InnerVolumeSpecName "kube-api-access-kqfg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:25.748738 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:25.748717    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-maas-api-xks-ca" pod "2b31c84e-3dbd-4dfd-a807-9f9019fa464c" (UID: "2b31c84e-3dbd-4dfd-a807-9f9019fa464c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:56:25.844602 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:25.844547    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqfg7\" (UniqueName: \"kubernetes.io/projected/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-kube-api-access-kqfg7\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:25.844909 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:25.844888    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b31c84e-3dbd-4dfd-a807-9f9019fa464c-maas-api-xks-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:26.226082 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:26.225999    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-f5xfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:56:26.226414 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:26.226375    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb" podUID="98ec8c1c-49db-492e-958e-fabe09d044fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:26.352090 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:26.352063    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"
Nov 03 19:56:26.352275 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:26.352102    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds" event={"ID":"2b31c84e-3dbd-4dfd-a807-9f9019fa464c","Type":"ContainerDied","Data":"0d495de8589758d428edc48e68c775a270d3187c8dec9616c6627b6afe6ee83f"}
Nov 03 19:56:26.352275 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:26.352151    5237 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d495de8589758d428edc48e68c775a270d3187c8dec9616c6627b6afe6ee83f"
Nov 03 19:56:30.726294 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:30.726256    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:32.794394 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:32.794352    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:43.254755 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:56:43.254713    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:56:43.757916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.757876    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"]
Nov 03 19:56:43.758265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.758251    5237 cpu_manager.go:414] "containerMap: removing container" podUID="2b31c84e-3dbd-4dfd-a807-9f9019fa464c" containerName="revoke-keys"
Nov 03 19:56:43.758313 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.758268    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b31c84e-3dbd-4dfd-a807-9f9019fa464c" containerName="revoke-keys"
Nov 03 19:56:43.758356 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.758349    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b31c84e-3dbd-4dfd-a807-9f9019fa464c" containerName="revoke-keys"
Nov 03 19:56:43.761511 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.761492    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"
Nov 03 19:56:43.766949 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.766923    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:56:43.768224 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.768199    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:56:43.772257 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.772234    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"]
Nov 03 19:56:43.805455 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.805417    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:56:43.805646 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.805556    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\""
Nov 03 19:56:43.809509 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.809477    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c79580-7caf-4796-a47d-a01f6a16fb73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-wqdhh\" (UID: \"98c79580-7caf-4796-a47d-a01f6a16fb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"
Nov 03 19:56:43.809601 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.809582    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6cbp\" (UniqueName: \"kubernetes.io/projected/98c79580-7caf-4796-a47d-a01f6a16fb73-kube-api-access-v6cbp\") pod \"maas-api-revoke-sub-ae594979-wqdhh\" (UID: \"98c79580-7caf-4796-a47d-a01f6a16fb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"
Nov 03 19:56:43.809754 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.809731    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c79580-7caf-4796-a47d-a01f6a16fb73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-wqdhh\" (UID: \"98c79580-7caf-4796-a47d-a01f6a16fb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"
Nov 03 19:56:43.911253 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.911211    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c79580-7caf-4796-a47d-a01f6a16fb73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-wqdhh\" (UID: \"98c79580-7caf-4796-a47d-a01f6a16fb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"
Nov 03 19:56:43.911436 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.911275    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c79580-7caf-4796-a47d-a01f6a16fb73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-wqdhh\" (UID: \"98c79580-7caf-4796-a47d-a01f6a16fb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"
Nov 03 19:56:43.911436 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.911300    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6cbp\" (UniqueName: \"kubernetes.io/projected/98c79580-7caf-4796-a47d-a01f6a16fb73-kube-api-access-v6cbp\") pod \"maas-api-revoke-sub-ae594979-wqdhh\" (UID: \"98c79580-7caf-4796-a47d-a01f6a16fb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"
Nov 03 19:56:43.912071 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.912050    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c79580-7caf-4796-a47d-a01f6a16fb73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-wqdhh\" (UID: \"98c79580-7caf-4796-a47d-a01f6a16fb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"
Nov 03 19:56:43.913605 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.913586    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c79580-7caf-4796-a47d-a01f6a16fb73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-wqdhh\" (UID: \"98c79580-7caf-4796-a47d-a01f6a16fb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"
Nov 03 19:56:43.923977 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:43.923948    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6cbp\" (UniqueName: \"kubernetes.io/projected/98c79580-7caf-4796-a47d-a01f6a16fb73-kube-api-access-v6cbp\") pod \"maas-api-revoke-sub-ae594979-wqdhh\" (UID: \"98c79580-7caf-4796-a47d-a01f6a16fb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"
Nov 03 19:56:44.075424 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:44.075384    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"
Nov 03 19:56:44.351974 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:44.351895    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"]
Nov 03 19:56:44.357410 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:56:44.357381    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c79580_7caf_4796_a47d_a01f6a16fb73.slice/crio-3dc5014876620b49b6bb1bd9607a51778b5d9dc443d97a3bae292f85db4f77d1 WatchSource:0}: Error finding container 3dc5014876620b49b6bb1bd9607a51778b5d9dc443d97a3bae292f85db4f77d1: Status 404 returned error can't find the container with id 3dc5014876620b49b6bb1bd9607a51778b5d9dc443d97a3bae292f85db4f77d1
Nov 03 19:56:44.450792 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:44.450752    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh" event={"ID":"98c79580-7caf-4796-a47d-a01f6a16fb73","Type":"ContainerStarted","Data":"3dc5014876620b49b6bb1bd9607a51778b5d9dc443d97a3bae292f85db4f77d1"}
Nov 03 19:56:45.456703 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:45.456610    5237 generic.go:356] "Generic (PLEG): container finished" podID="98c79580-7caf-4796-a47d-a01f6a16fb73" containerID="b48ccfc21d0a2d19a6fddc26cad9c85682779e30aade23e2468c40504e24b7d2" exitCode=0
Nov 03 19:56:45.457110 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:45.456701    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh" event={"ID":"98c79580-7caf-4796-a47d-a01f6a16fb73","Type":"ContainerDied","Data":"b48ccfc21d0a2d19a6fddc26cad9c85682779e30aade23e2468c40504e24b7d2"}
Nov 03 19:56:46.588004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:46.587977    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"
Nov 03 19:56:46.634382 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:46.634348    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98c79580-7caf-4796-a47d-a01f6a16fb73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c79580-7caf-4796-a47d-a01f6a16fb73-maas-api-openshift-ca\") pod \"98c79580-7caf-4796-a47d-a01f6a16fb73\" (UID: \"98c79580-7caf-4796-a47d-a01f6a16fb73\") "
Nov 03 19:56:46.634587 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:46.634443    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98c79580-7caf-4796-a47d-a01f6a16fb73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c79580-7caf-4796-a47d-a01f6a16fb73-maas-api-xks-ca\") pod \"98c79580-7caf-4796-a47d-a01f6a16fb73\" (UID: \"98c79580-7caf-4796-a47d-a01f6a16fb73\") "
Nov 03 19:56:46.634587 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:46.634501    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c79580-7caf-4796-a47d-a01f6a16fb73-kube-api-access-v6cbp\" (UniqueName: \"kubernetes.io/projected/98c79580-7caf-4796-a47d-a01f6a16fb73-kube-api-access-v6cbp\") pod \"98c79580-7caf-4796-a47d-a01f6a16fb73\" (UID: \"98c79580-7caf-4796-a47d-a01f6a16fb73\") "
Nov 03 19:56:46.634900 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:46.634826    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c79580-7caf-4796-a47d-a01f6a16fb73-maas-api-openshift-ca" pod "98c79580-7caf-4796-a47d-a01f6a16fb73" (UID: "98c79580-7caf-4796-a47d-a01f6a16fb73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:56:46.636562 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:46.636538    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98c79580-7caf-4796-a47d-a01f6a16fb73-maas-api-xks-ca" pod "98c79580-7caf-4796-a47d-a01f6a16fb73" (UID: "98c79580-7caf-4796-a47d-a01f6a16fb73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:56:46.636689 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:46.636669    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c79580-7caf-4796-a47d-a01f6a16fb73-kube-api-access-v6cbp" pod "98c79580-7caf-4796-a47d-a01f6a16fb73" (UID: "98c79580-7caf-4796-a47d-a01f6a16fb73"). InnerVolumeSpecName "kube-api-access-v6cbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:46.735475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:46.735381    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c79580-7caf-4796-a47d-a01f6a16fb73-maas-api-xks-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:46.735475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:46.735415    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6cbp\" (UniqueName: \"kubernetes.io/projected/98c79580-7caf-4796-a47d-a01f6a16fb73-kube-api-access-v6cbp\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:46.735475 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:46.735425    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c79580-7caf-4796-a47d-a01f6a16fb73-maas-api-openshift-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:47.467020 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:47.466989    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"
Nov 03 19:56:47.467214 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:47.466988    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh" event={"ID":"98c79580-7caf-4796-a47d-a01f6a16fb73","Type":"ContainerDied","Data":"3dc5014876620b49b6bb1bd9607a51778b5d9dc443d97a3bae292f85db4f77d1"}
Nov 03 19:56:47.467214 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:56:47.467111    5237 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dc5014876620b49b6bb1bd9607a51778b5d9dc443d97a3bae292f85db4f77d1"
Nov 03 19:57:01.905540 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:01.905500    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"]
Nov 03 19:57:01.905963 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:01.905933    5237 cpu_manager.go:414] "containerMap: removing container" podUID="98c79580-7caf-4796-a47d-a01f6a16fb73" containerName="revoke-keys"
Nov 03 19:57:01.905963 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:01.905946    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c79580-7caf-4796-a47d-a01f6a16fb73" containerName="revoke-keys"
Nov 03 19:57:01.906046 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:01.906024    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c79580-7caf-4796-a47d-a01f6a16fb73" containerName="revoke-keys"
Nov 03 19:57:01.909247 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:01.909230    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"
Nov 03 19:57:01.913348 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:01.913320    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:57:01.913348 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:01.913335    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:57:01.914792 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:01.914770    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:57:01.914915 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:01.914816    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\""
Nov 03 19:57:01.919430 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:01.919401    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"]
Nov 03 19:57:01.977465 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:01.977424    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c4a7ac-0729-400f-99fb-a7bea0bf372b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-l7fg5\" (UID: \"88c4a7ac-0729-400f-99fb-a7bea0bf372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"
Nov 03 19:57:01.977682 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:01.977553    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w9lk\" (UniqueName: \"kubernetes.io/projected/88c4a7ac-0729-400f-99fb-a7bea0bf372b-kube-api-access-5w9lk\") pod \"maas-api-revoke-sub-a4aef7d7-l7fg5\" (UID: \"88c4a7ac-0729-400f-99fb-a7bea0bf372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"
Nov 03 19:57:01.977682 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:01.977593    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c4a7ac-0729-400f-99fb-a7bea0bf372b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-l7fg5\" (UID: \"88c4a7ac-0729-400f-99fb-a7bea0bf372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"
Nov 03 19:57:02.079022 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:02.078983    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w9lk\" (UniqueName: \"kubernetes.io/projected/88c4a7ac-0729-400f-99fb-a7bea0bf372b-kube-api-access-5w9lk\") pod \"maas-api-revoke-sub-a4aef7d7-l7fg5\" (UID: \"88c4a7ac-0729-400f-99fb-a7bea0bf372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"
Nov 03 19:57:02.079022 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:02.079029    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c4a7ac-0729-400f-99fb-a7bea0bf372b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-l7fg5\" (UID: \"88c4a7ac-0729-400f-99fb-a7bea0bf372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"
Nov 03 19:57:02.079265 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:02.079090    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c4a7ac-0729-400f-99fb-a7bea0bf372b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-l7fg5\" (UID: \"88c4a7ac-0729-400f-99fb-a7bea0bf372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"
Nov 03 19:57:02.079874 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:02.079831    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c4a7ac-0729-400f-99fb-a7bea0bf372b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-l7fg5\" (UID: \"88c4a7ac-0729-400f-99fb-a7bea0bf372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"
Nov 03 19:57:02.081488 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:02.081465    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c4a7ac-0729-400f-99fb-a7bea0bf372b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-l7fg5\" (UID: \"88c4a7ac-0729-400f-99fb-a7bea0bf372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"
Nov 03 19:57:02.091251 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:02.091225    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w9lk\" (UniqueName: \"kubernetes.io/projected/88c4a7ac-0729-400f-99fb-a7bea0bf372b-kube-api-access-5w9lk\") pod \"maas-api-revoke-sub-a4aef7d7-l7fg5\" (UID: \"88c4a7ac-0729-400f-99fb-a7bea0bf372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"
Nov 03 19:57:02.222752 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:02.222661    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"
Nov 03 19:57:02.498200 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:02.498171    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"]
Nov 03 19:57:02.500751 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:57:02.500709    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c4a7ac_0729_400f_99fb_a7bea0bf372b.slice/crio-7ab8d076f85cefe6016f5e1be8a67ff861fc714de58346d4cf96111e3720b5c2 WatchSource:0}: Error finding container 7ab8d076f85cefe6016f5e1be8a67ff861fc714de58346d4cf96111e3720b5c2: Status 404 returned error can't find the container with id 7ab8d076f85cefe6016f5e1be8a67ff861fc714de58346d4cf96111e3720b5c2
Nov 03 19:57:02.559447 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:02.559411    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5" event={"ID":"88c4a7ac-0729-400f-99fb-a7bea0bf372b","Type":"ContainerStarted","Data":"7ab8d076f85cefe6016f5e1be8a67ff861fc714de58346d4cf96111e3720b5c2"}
Nov 03 19:57:03.566809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:03.566765    5237 generic.go:356] "Generic (PLEG): container finished" podID="88c4a7ac-0729-400f-99fb-a7bea0bf372b" containerID="75a0664ea501c902463e00e8d9960062b01dd7058cc1300dd84e49a5204b2865" exitCode=0
Nov 03 19:57:03.567270 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:03.566861    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5" event={"ID":"88c4a7ac-0729-400f-99fb-a7bea0bf372b","Type":"ContainerDied","Data":"75a0664ea501c902463e00e8d9960062b01dd7058cc1300dd84e49a5204b2865"}
Nov 03 19:57:04.697917 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:04.697879    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"
Nov 03 19:57:04.802764 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:04.802727    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88c4a7ac-0729-400f-99fb-a7bea0bf372b-kube-api-access-5w9lk\" (UniqueName: \"kubernetes.io/projected/88c4a7ac-0729-400f-99fb-a7bea0bf372b-kube-api-access-5w9lk\") pod \"88c4a7ac-0729-400f-99fb-a7bea0bf372b\" (UID: \"88c4a7ac-0729-400f-99fb-a7bea0bf372b\") "
Nov 03 19:57:04.803008 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:04.802796    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88c4a7ac-0729-400f-99fb-a7bea0bf372b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c4a7ac-0729-400f-99fb-a7bea0bf372b-maas-api-openshift-ca\") pod \"88c4a7ac-0729-400f-99fb-a7bea0bf372b\" (UID: \"88c4a7ac-0729-400f-99fb-a7bea0bf372b\") "
Nov 03 19:57:04.803008 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:04.802881    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88c4a7ac-0729-400f-99fb-a7bea0bf372b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c4a7ac-0729-400f-99fb-a7bea0bf372b-maas-api-xks-ca\") pod \"88c4a7ac-0729-400f-99fb-a7bea0bf372b\" (UID: \"88c4a7ac-0729-400f-99fb-a7bea0bf372b\") "
Nov 03 19:57:04.803233 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:04.803207    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c4a7ac-0729-400f-99fb-a7bea0bf372b-maas-api-openshift-ca" pod "88c4a7ac-0729-400f-99fb-a7bea0bf372b" (UID: "88c4a7ac-0729-400f-99fb-a7bea0bf372b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:04.804933 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:04.804911    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c4a7ac-0729-400f-99fb-a7bea0bf372b-kube-api-access-5w9lk" pod "88c4a7ac-0729-400f-99fb-a7bea0bf372b" (UID: "88c4a7ac-0729-400f-99fb-a7bea0bf372b"). InnerVolumeSpecName "kube-api-access-5w9lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:04.805002 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:04.804927    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c4a7ac-0729-400f-99fb-a7bea0bf372b-maas-api-xks-ca" pod "88c4a7ac-0729-400f-99fb-a7bea0bf372b" (UID: "88c4a7ac-0729-400f-99fb-a7bea0bf372b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:57:04.903868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:04.903756    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w9lk\" (UniqueName: \"kubernetes.io/projected/88c4a7ac-0729-400f-99fb-a7bea0bf372b-kube-api-access-5w9lk\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:04.903868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:04.903788    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c4a7ac-0729-400f-99fb-a7bea0bf372b-maas-api-openshift-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:04.903868 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:04.903799    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c4a7ac-0729-400f-99fb-a7bea0bf372b-maas-api-xks-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:05.575909 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:05.575877    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"
Nov 03 19:57:05.576127 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:05.575901    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5" event={"ID":"88c4a7ac-0729-400f-99fb-a7bea0bf372b","Type":"ContainerDied","Data":"7ab8d076f85cefe6016f5e1be8a67ff861fc714de58346d4cf96111e3720b5c2"}
Nov 03 19:57:05.576127 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:05.575960    5237 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ab8d076f85cefe6016f5e1be8a67ff861fc714de58346d4cf96111e3720b5c2"
Nov 03 19:57:13.017331 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:13.017288    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:13.260860 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:57:13.259927    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:57:20.529629 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.529592    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"]
Nov 03 19:57:20.530216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.529970    5237 cpu_manager.go:414] "containerMap: removing container" podUID="88c4a7ac-0729-400f-99fb-a7bea0bf372b" containerName="revoke-keys"
Nov 03 19:57:20.530216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.529985    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88c4a7ac-0729-400f-99fb-a7bea0bf372b" containerName="revoke-keys"
Nov 03 19:57:20.530216 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.530115    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="88c4a7ac-0729-400f-99fb-a7bea0bf372b" containerName="revoke-keys"
Nov 03 19:57:20.533051 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.533026    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"
Nov 03 19:57:20.537363 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.537340    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:57:20.538664 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.538648    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:57:20.538771 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.538757    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:57:20.538876 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.538859    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\""
Nov 03 19:57:20.543456 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.543386    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"]
Nov 03 19:57:20.576011 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.575978    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr2q6\" (UniqueName: \"kubernetes.io/projected/30ed0d74-db0f-400a-930c-e889d9776754-kube-api-access-qr2q6\") pod \"maas-api-revoke-sub-79805ca9-xp2kr\" (UID: \"30ed0d74-db0f-400a-930c-e889d9776754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"
Nov 03 19:57:20.576179 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.576118    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ed0d74-db0f-400a-930c-e889d9776754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xp2kr\" (UID: \"30ed0d74-db0f-400a-930c-e889d9776754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"
Nov 03 19:57:20.576221 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.576174    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ed0d74-db0f-400a-930c-e889d9776754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xp2kr\" (UID: \"30ed0d74-db0f-400a-930c-e889d9776754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"
Nov 03 19:57:20.677194 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.677151    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ed0d74-db0f-400a-930c-e889d9776754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xp2kr\" (UID: \"30ed0d74-db0f-400a-930c-e889d9776754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"
Nov 03 19:57:20.677194 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.677196    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ed0d74-db0f-400a-930c-e889d9776754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xp2kr\" (UID: \"30ed0d74-db0f-400a-930c-e889d9776754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"
Nov 03 19:57:20.677429 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.677218    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr2q6\" (UniqueName: \"kubernetes.io/projected/30ed0d74-db0f-400a-930c-e889d9776754-kube-api-access-qr2q6\") pod \"maas-api-revoke-sub-79805ca9-xp2kr\" (UID: \"30ed0d74-db0f-400a-930c-e889d9776754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"
Nov 03 19:57:20.677813 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.677779    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ed0d74-db0f-400a-930c-e889d9776754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xp2kr\" (UID: \"30ed0d74-db0f-400a-930c-e889d9776754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"
Nov 03 19:57:20.679558 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.679531    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ed0d74-db0f-400a-930c-e889d9776754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xp2kr\" (UID: \"30ed0d74-db0f-400a-930c-e889d9776754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"
Nov 03 19:57:20.689403 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.689370    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr2q6\" (UniqueName: \"kubernetes.io/projected/30ed0d74-db0f-400a-930c-e889d9776754-kube-api-access-qr2q6\") pod \"maas-api-revoke-sub-79805ca9-xp2kr\" (UID: \"30ed0d74-db0f-400a-930c-e889d9776754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"
Nov 03 19:57:20.847117 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:20.847075    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"
Nov 03 19:57:21.121309 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:21.121287    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"]
Nov 03 19:57:21.123656 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:57:21.123623    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ed0d74_db0f_400a_930c_e889d9776754.slice/crio-b53d17e68b41eb943b8ebf2b5f963312a561fe5b5010fe34dfc8a8ba5740ccee WatchSource:0}: Error finding container b53d17e68b41eb943b8ebf2b5f963312a561fe5b5010fe34dfc8a8ba5740ccee: Status 404 returned error can't find the container with id b53d17e68b41eb943b8ebf2b5f963312a561fe5b5010fe34dfc8a8ba5740ccee
Nov 03 19:57:21.656323 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:21.656230    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr" event={"ID":"30ed0d74-db0f-400a-930c-e889d9776754","Type":"ContainerStarted","Data":"9cb77e6451b655637ce2362c21fe706be1d86748edc81e0b16687168071d691a"}
Nov 03 19:57:21.656323 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:21.656300    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr" event={"ID":"30ed0d74-db0f-400a-930c-e889d9776754","Type":"ContainerStarted","Data":"b53d17e68b41eb943b8ebf2b5f963312a561fe5b5010fe34dfc8a8ba5740ccee"}
Nov 03 19:57:21.676541 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:21.676477    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr" podStartSLOduration=1.676459617 podStartE2EDuration="1.676459617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:21.674866381 +0000 UTC m=+1797.929173610" watchObservedRunningTime="2026-11-03 19:57:21.676459617 +0000 UTC m=+1797.930766843"
Nov 03 19:57:22.661639 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:22.661607    5237 generic.go:356] "Generic (PLEG): container finished" podID="30ed0d74-db0f-400a-930c-e889d9776754" containerID="9cb77e6451b655637ce2362c21fe706be1d86748edc81e0b16687168071d691a" exitCode=0
Nov 03 19:57:22.662044 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:22.661702    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr" event={"ID":"30ed0d74-db0f-400a-930c-e889d9776754","Type":"ContainerDied","Data":"9cb77e6451b655637ce2362c21fe706be1d86748edc81e0b16687168071d691a"}
Nov 03 19:57:23.795317 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:23.795293    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"
Nov 03 19:57:23.906464 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:23.906431    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30ed0d74-db0f-400a-930c-e889d9776754-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ed0d74-db0f-400a-930c-e889d9776754-maas-api-xks-ca\") pod \"30ed0d74-db0f-400a-930c-e889d9776754\" (UID: \"30ed0d74-db0f-400a-930c-e889d9776754\") "
Nov 03 19:57:23.906657 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:23.906576    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ed0d74-db0f-400a-930c-e889d9776754-kube-api-access-qr2q6\" (UniqueName: \"kubernetes.io/projected/30ed0d74-db0f-400a-930c-e889d9776754-kube-api-access-qr2q6\") pod \"30ed0d74-db0f-400a-930c-e889d9776754\" (UID: \"30ed0d74-db0f-400a-930c-e889d9776754\") "
Nov 03 19:57:23.906657 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:23.906607    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30ed0d74-db0f-400a-930c-e889d9776754-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ed0d74-db0f-400a-930c-e889d9776754-maas-api-openshift-ca\") pod \"30ed0d74-db0f-400a-930c-e889d9776754\" (UID: \"30ed0d74-db0f-400a-930c-e889d9776754\") "
Nov 03 19:57:23.907056 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:23.907023    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ed0d74-db0f-400a-930c-e889d9776754-maas-api-openshift-ca" pod "30ed0d74-db0f-400a-930c-e889d9776754" (UID: "30ed0d74-db0f-400a-930c-e889d9776754"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:23.908505 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:23.908479    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ed0d74-db0f-400a-930c-e889d9776754-maas-api-xks-ca" pod "30ed0d74-db0f-400a-930c-e889d9776754" (UID: "30ed0d74-db0f-400a-930c-e889d9776754"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:57:23.908606 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:23.908543    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ed0d74-db0f-400a-930c-e889d9776754-kube-api-access-qr2q6" pod "30ed0d74-db0f-400a-930c-e889d9776754" (UID: "30ed0d74-db0f-400a-930c-e889d9776754"). InnerVolumeSpecName "kube-api-access-qr2q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:24.007439 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:24.007353    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ed0d74-db0f-400a-930c-e889d9776754-maas-api-xks-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:24.007439 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:24.007382    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr2q6\" (UniqueName: \"kubernetes.io/projected/30ed0d74-db0f-400a-930c-e889d9776754-kube-api-access-qr2q6\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:24.007439 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:24.007392    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ed0d74-db0f-400a-930c-e889d9776754-maas-api-openshift-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:24.672641 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:24.672601    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr" event={"ID":"30ed0d74-db0f-400a-930c-e889d9776754","Type":"ContainerDied","Data":"b53d17e68b41eb943b8ebf2b5f963312a561fe5b5010fe34dfc8a8ba5740ccee"}
Nov 03 19:57:24.672641 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:24.672635    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"
Nov 03 19:57:24.672942 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:24.672653    5237 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b53d17e68b41eb943b8ebf2b5f963312a561fe5b5010fe34dfc8a8ba5740ccee"
Nov 03 19:57:31.018733 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:31.018692    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:31.168790 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:31.168747    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:38.304642 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.304556    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"]
Nov 03 19:57:38.305079 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.304935    5237 cpu_manager.go:414] "containerMap: removing container" podUID="30ed0d74-db0f-400a-930c-e889d9776754" containerName="revoke-keys"
Nov 03 19:57:38.305079 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.304947    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ed0d74-db0f-400a-930c-e889d9776754" containerName="revoke-keys"
Nov 03 19:57:38.305079 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.305011    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ed0d74-db0f-400a-930c-e889d9776754" containerName="revoke-keys"
Nov 03 19:57:38.308129 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.308107    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"
Nov 03 19:57:38.312383 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.312349    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:57:38.316171 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.316143    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\""
Nov 03 19:57:38.318135 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.318111    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"]
Nov 03 19:57:38.358744 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.358711    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142001c1-ed1e-430f-8fdf-336807ccab3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wmx5j\" (UID: \"142001c1-ed1e-430f-8fdf-336807ccab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"
Nov 03 19:57:38.358951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.358771    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142001c1-ed1e-430f-8fdf-336807ccab3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wmx5j\" (UID: \"142001c1-ed1e-430f-8fdf-336807ccab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"
Nov 03 19:57:38.358951 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.358879    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glpfk\" (UniqueName: \"kubernetes.io/projected/142001c1-ed1e-430f-8fdf-336807ccab3c-kube-api-access-glpfk\") pod \"maas-api-revoke-sub-0b7e8edb-wmx5j\" (UID: \"142001c1-ed1e-430f-8fdf-336807ccab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"
Nov 03 19:57:38.387781 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.387755    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:57:38.387781 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.387769    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:57:38.460693 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.460461    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142001c1-ed1e-430f-8fdf-336807ccab3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wmx5j\" (UID: \"142001c1-ed1e-430f-8fdf-336807ccab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"
Nov 03 19:57:38.460693 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.460528    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142001c1-ed1e-430f-8fdf-336807ccab3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wmx5j\" (UID: \"142001c1-ed1e-430f-8fdf-336807ccab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"
Nov 03 19:57:38.460693 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.460575    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glpfk\" (UniqueName: \"kubernetes.io/projected/142001c1-ed1e-430f-8fdf-336807ccab3c-kube-api-access-glpfk\") pod \"maas-api-revoke-sub-0b7e8edb-wmx5j\" (UID: \"142001c1-ed1e-430f-8fdf-336807ccab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"
Nov 03 19:57:38.461377 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.461351    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142001c1-ed1e-430f-8fdf-336807ccab3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wmx5j\" (UID: \"142001c1-ed1e-430f-8fdf-336807ccab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"
Nov 03 19:57:38.465037 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.465013    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142001c1-ed1e-430f-8fdf-336807ccab3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wmx5j\" (UID: \"142001c1-ed1e-430f-8fdf-336807ccab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"
Nov 03 19:57:38.478338 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.478313    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glpfk\" (UniqueName: \"kubernetes.io/projected/142001c1-ed1e-430f-8fdf-336807ccab3c-kube-api-access-glpfk\") pod \"maas-api-revoke-sub-0b7e8edb-wmx5j\" (UID: \"142001c1-ed1e-430f-8fdf-336807ccab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"
Nov 03 19:57:38.621778 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.621688    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"
Nov 03 19:57:38.895725 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:38.895648    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"]
Nov 03 19:57:38.899609 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:57:38.899580    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142001c1_ed1e_430f_8fdf_336807ccab3c.slice/crio-e371b411ebb73245b0251ca21a49f06f7acc590ba827801b1ac82dfddf64e92e WatchSource:0}: Error finding container e371b411ebb73245b0251ca21a49f06f7acc590ba827801b1ac82dfddf64e92e: Status 404 returned error can't find the container with id e371b411ebb73245b0251ca21a49f06f7acc590ba827801b1ac82dfddf64e92e
Nov 03 19:57:39.758542 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:39.758498    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j" event={"ID":"142001c1-ed1e-430f-8fdf-336807ccab3c","Type":"ContainerStarted","Data":"ac2acfa47b1b7f1712c66b22a6d15836e6770c2057ea3ac857dbac0f979455ca"}
Nov 03 19:57:39.758978 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:39.758600    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j" event={"ID":"142001c1-ed1e-430f-8fdf-336807ccab3c","Type":"ContainerStarted","Data":"e371b411ebb73245b0251ca21a49f06f7acc590ba827801b1ac82dfddf64e92e"}
Nov 03 19:57:39.789249 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:39.789198    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j" podStartSLOduration=1.789184495 podStartE2EDuration="1.789184495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:39.788646115 +0000 UTC m=+1816.042953344" watchObservedRunningTime="2026-11-03 19:57:39.789184495 +0000 UTC m=+1816.043491719"
Nov 03 19:57:40.764906 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:40.764867    5237 generic.go:356] "Generic (PLEG): container finished" podID="142001c1-ed1e-430f-8fdf-336807ccab3c" containerID="ac2acfa47b1b7f1712c66b22a6d15836e6770c2057ea3ac857dbac0f979455ca" exitCode=0
Nov 03 19:57:40.764906 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:40.764905    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j" event={"ID":"142001c1-ed1e-430f-8fdf-336807ccab3c","Type":"ContainerDied","Data":"ac2acfa47b1b7f1712c66b22a6d15836e6770c2057ea3ac857dbac0f979455ca"}
Nov 03 19:57:41.898557 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:41.898529    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"
Nov 03 19:57:41.994315 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:41.994258    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/142001c1-ed1e-430f-8fdf-336807ccab3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142001c1-ed1e-430f-8fdf-336807ccab3c-maas-api-xks-ca\") pod \"142001c1-ed1e-430f-8fdf-336807ccab3c\" (UID: \"142001c1-ed1e-430f-8fdf-336807ccab3c\") "
Nov 03 19:57:41.994517 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:41.994366    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/142001c1-ed1e-430f-8fdf-336807ccab3c-kube-api-access-glpfk\" (UniqueName: \"kubernetes.io/projected/142001c1-ed1e-430f-8fdf-336807ccab3c-kube-api-access-glpfk\") pod \"142001c1-ed1e-430f-8fdf-336807ccab3c\" (UID: \"142001c1-ed1e-430f-8fdf-336807ccab3c\") "
Nov 03 19:57:41.994517 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:41.994406    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/142001c1-ed1e-430f-8fdf-336807ccab3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142001c1-ed1e-430f-8fdf-336807ccab3c-maas-api-openshift-ca\") pod \"142001c1-ed1e-430f-8fdf-336807ccab3c\" (UID: \"142001c1-ed1e-430f-8fdf-336807ccab3c\") "
Nov 03 19:57:41.994815 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:41.994786    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/142001c1-ed1e-430f-8fdf-336807ccab3c-maas-api-openshift-ca" pod "142001c1-ed1e-430f-8fdf-336807ccab3c" (UID: "142001c1-ed1e-430f-8fdf-336807ccab3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:41.996329 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:41.996312    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/142001c1-ed1e-430f-8fdf-336807ccab3c-maas-api-xks-ca" pod "142001c1-ed1e-430f-8fdf-336807ccab3c" (UID: "142001c1-ed1e-430f-8fdf-336807ccab3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:57:41.996467 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:41.996444    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142001c1-ed1e-430f-8fdf-336807ccab3c-kube-api-access-glpfk" pod "142001c1-ed1e-430f-8fdf-336807ccab3c" (UID: "142001c1-ed1e-430f-8fdf-336807ccab3c"). InnerVolumeSpecName "kube-api-access-glpfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:42.095050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:42.095010    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142001c1-ed1e-430f-8fdf-336807ccab3c-maas-api-xks-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:42.095050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:42.095041    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glpfk\" (UniqueName: \"kubernetes.io/projected/142001c1-ed1e-430f-8fdf-336807ccab3c-kube-api-access-glpfk\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:42.095050 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:42.095052    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142001c1-ed1e-430f-8fdf-336807ccab3c-maas-api-openshift-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:42.775311 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:42.775277    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j" event={"ID":"142001c1-ed1e-430f-8fdf-336807ccab3c","Type":"ContainerDied","Data":"e371b411ebb73245b0251ca21a49f06f7acc590ba827801b1ac82dfddf64e92e"}
Nov 03 19:57:42.775311 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:42.775312    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"
Nov 03 19:57:42.775529 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:42.775327    5237 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e371b411ebb73245b0251ca21a49f06f7acc590ba827801b1ac82dfddf64e92e"
Nov 03 19:57:43.254310 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:57:43.254274    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:57:48.177400 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:48.177360    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:48.309311 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:48.309278    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:48.965911 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:48.965865    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"]
Nov 03 19:57:48.966297 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:48.966279    5237 cpu_manager.go:414] "containerMap: removing container" podUID="142001c1-ed1e-430f-8fdf-336807ccab3c" containerName="revoke-keys"
Nov 03 19:57:48.966377 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:48.966300    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="142001c1-ed1e-430f-8fdf-336807ccab3c" containerName="revoke-keys"
Nov 03 19:57:48.966453 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:48.966418    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="142001c1-ed1e-430f-8fdf-336807ccab3c" containerName="revoke-keys"
Nov 03 19:57:48.969195 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:48.969176    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"
Nov 03 19:57:48.973880 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:48.973859    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:57:48.975282 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:48.975266    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:57:48.979416 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:48.979390    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"]
Nov 03 19:57:49.041631 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:49.041595    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:57:49.041631 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:49.041623    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\""
Nov 03 19:57:49.053068 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:49.053028    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf9411f-0622-424d-8672-c5de171d3365-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mm7k2\" (UID: \"edf9411f-0622-424d-8672-c5de171d3365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"
Nov 03 19:57:49.053198 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:49.053162    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz6jt\" (UniqueName: \"kubernetes.io/projected/edf9411f-0622-424d-8672-c5de171d3365-kube-api-access-nz6jt\") pod \"maas-api-revoke-sub-3d4156f1-mm7k2\" (UID: \"edf9411f-0622-424d-8672-c5de171d3365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"
Nov 03 19:57:49.053198 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:49.053190    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf9411f-0622-424d-8672-c5de171d3365-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mm7k2\" (UID: \"edf9411f-0622-424d-8672-c5de171d3365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"
Nov 03 19:57:49.154037 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:49.153988    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz6jt\" (UniqueName: \"kubernetes.io/projected/edf9411f-0622-424d-8672-c5de171d3365-kube-api-access-nz6jt\") pod \"maas-api-revoke-sub-3d4156f1-mm7k2\" (UID: \"edf9411f-0622-424d-8672-c5de171d3365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"
Nov 03 19:57:49.154037 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:49.154046    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf9411f-0622-424d-8672-c5de171d3365-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mm7k2\" (UID: \"edf9411f-0622-424d-8672-c5de171d3365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"
Nov 03 19:57:49.154318 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:49.154118    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf9411f-0622-424d-8672-c5de171d3365-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mm7k2\" (UID: \"edf9411f-0622-424d-8672-c5de171d3365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"
Nov 03 19:57:49.154780 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:49.154751    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf9411f-0622-424d-8672-c5de171d3365-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mm7k2\" (UID: \"edf9411f-0622-424d-8672-c5de171d3365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"
Nov 03 19:57:49.156549 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:49.156525    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf9411f-0622-424d-8672-c5de171d3365-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mm7k2\" (UID: \"edf9411f-0622-424d-8672-c5de171d3365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"
Nov 03 19:57:49.166896 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:49.166865    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz6jt\" (UniqueName: \"kubernetes.io/projected/edf9411f-0622-424d-8672-c5de171d3365-kube-api-access-nz6jt\") pod \"maas-api-revoke-sub-3d4156f1-mm7k2\" (UID: \"edf9411f-0622-424d-8672-c5de171d3365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"
Nov 03 19:57:49.281640 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:49.281560    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"
Nov 03 19:57:49.555689 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:49.555620    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"]
Nov 03 19:57:49.558207 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:57:49.558181    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf9411f_0622_424d_8672_c5de171d3365.slice/crio-801cc4b1d423f3a73596004b5c845c7c8868e19e80bd79d4dec1d2cc38dd1f9b WatchSource:0}: Error finding container 801cc4b1d423f3a73596004b5c845c7c8868e19e80bd79d4dec1d2cc38dd1f9b: Status 404 returned error can't find the container with id 801cc4b1d423f3a73596004b5c845c7c8868e19e80bd79d4dec1d2cc38dd1f9b
Nov 03 19:57:49.807853 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:49.807755    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2" event={"ID":"edf9411f-0622-424d-8672-c5de171d3365","Type":"ContainerStarted","Data":"801cc4b1d423f3a73596004b5c845c7c8868e19e80bd79d4dec1d2cc38dd1f9b"}
Nov 03 19:57:50.813448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:50.813408    5237 generic.go:356] "Generic (PLEG): container finished" podID="edf9411f-0622-424d-8672-c5de171d3365" containerID="486a03aada27a948a4c25c5313abacf4876e319d83ed015ddd705e530de5091f" exitCode=0
Nov 03 19:57:50.813942 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:50.813470    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2" event={"ID":"edf9411f-0622-424d-8672-c5de171d3365","Type":"ContainerDied","Data":"486a03aada27a948a4c25c5313abacf4876e319d83ed015ddd705e530de5091f"}
Nov 03 19:57:51.946788 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:51.946763    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"
Nov 03 19:57:52.083471 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:52.083376    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edf9411f-0622-424d-8672-c5de171d3365-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf9411f-0622-424d-8672-c5de171d3365-maas-api-openshift-ca\") pod \"edf9411f-0622-424d-8672-c5de171d3365\" (UID: \"edf9411f-0622-424d-8672-c5de171d3365\") "
Nov 03 19:57:52.083471 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:52.083438    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edf9411f-0622-424d-8672-c5de171d3365-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf9411f-0622-424d-8672-c5de171d3365-maas-api-xks-ca\") pod \"edf9411f-0622-424d-8672-c5de171d3365\" (UID: \"edf9411f-0622-424d-8672-c5de171d3365\") "
Nov 03 19:57:52.083708 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:52.083508    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf9411f-0622-424d-8672-c5de171d3365-kube-api-access-nz6jt\" (UniqueName: \"kubernetes.io/projected/edf9411f-0622-424d-8672-c5de171d3365-kube-api-access-nz6jt\") pod \"edf9411f-0622-424d-8672-c5de171d3365\" (UID: \"edf9411f-0622-424d-8672-c5de171d3365\") "
Nov 03 19:57:52.083789 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:52.083758    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf9411f-0622-424d-8672-c5de171d3365-maas-api-openshift-ca" pod "edf9411f-0622-424d-8672-c5de171d3365" (UID: "edf9411f-0622-424d-8672-c5de171d3365"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:52.085595 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:52.085567    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf9411f-0622-424d-8672-c5de171d3365-maas-api-xks-ca" pod "edf9411f-0622-424d-8672-c5de171d3365" (UID: "edf9411f-0622-424d-8672-c5de171d3365"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:57:52.085595 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:52.085572    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf9411f-0622-424d-8672-c5de171d3365-kube-api-access-nz6jt" pod "edf9411f-0622-424d-8672-c5de171d3365" (UID: "edf9411f-0622-424d-8672-c5de171d3365"). InnerVolumeSpecName "kube-api-access-nz6jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:52.184426 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:52.184388    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf9411f-0622-424d-8672-c5de171d3365-maas-api-openshift-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:52.184426 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:52.184424    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf9411f-0622-424d-8672-c5de171d3365-maas-api-xks-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:52.184426 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:52.184433    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz6jt\" (UniqueName: \"kubernetes.io/projected/edf9411f-0622-424d-8672-c5de171d3365-kube-api-access-nz6jt\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:52.823768 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:52.823742    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"
Nov 03 19:57:52.823768 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:52.823754    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2" event={"ID":"edf9411f-0622-424d-8672-c5de171d3365","Type":"ContainerDied","Data":"801cc4b1d423f3a73596004b5c845c7c8868e19e80bd79d4dec1d2cc38dd1f9b"}
Nov 03 19:57:52.824015 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:57:52.823804    5237 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="801cc4b1d423f3a73596004b5c845c7c8868e19e80bd79d4dec1d2cc38dd1f9b"
Nov 03 19:58:01.583897 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:01.583857    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:01.733448 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:01.733410    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:05.347076 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:05.347027    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-t9ft8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:58:05.347496 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:05.347106    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8" podUID="c8d65667-e957-4334-a43a-71d03f1d1000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:05.407002 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:05.406961    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-bqzwl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:58:05.407177 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:05.407024    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl" podUID="f5da6be6-a4c0-40f5-8f07-d3de1d188394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:06.321250 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:06.321187    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:06.453524 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:06.453477    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:10.242603 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.242567    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"]
Nov 03 19:58:10.251126 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.251100    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mm7k2"]
Nov 03 19:58:10.273419 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.273383    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"]
Nov 03 19:58:10.273926 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.273909    5237 cpu_manager.go:414] "containerMap: removing container" podUID="edf9411f-0622-424d-8672-c5de171d3365" containerName="revoke-keys"
Nov 03 19:58:10.273987 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.273929    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf9411f-0622-424d-8672-c5de171d3365" containerName="revoke-keys"
Nov 03 19:58:10.274051 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.274040    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf9411f-0622-424d-8672-c5de171d3365" containerName="revoke-keys"
Nov 03 19:58:10.284441 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.284417    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"
Nov 03 19:58:10.287156 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.287126    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"]
Nov 03 19:58:10.289235 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.289215    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\""
Nov 03 19:58:10.289888 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.289826    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qgrf6\" (UID: \"63c6ea74-7fa4-4b22-9d03-4c1b33efce39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"
Nov 03 19:58:10.290018 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.289982    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qgrf6\" (UID: \"63c6ea74-7fa4-4b22-9d03-4c1b33efce39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"
Nov 03 19:58:10.290018 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.290013    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6cbc\" (UniqueName: \"kubernetes.io/projected/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-kube-api-access-v6cbc\") pod \"maas-api-revoke-sub-3d4156f1-qgrf6\" (UID: \"63c6ea74-7fa4-4b22-9d03-4c1b33efce39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"
Nov 03 19:58:10.292194 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.292175    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:58:10.292565 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.292549    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:58:10.292692 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.292678    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:58:10.391030 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.390993    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qgrf6\" (UID: \"63c6ea74-7fa4-4b22-9d03-4c1b33efce39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"
Nov 03 19:58:10.391215 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.391074    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qgrf6\" (UID: \"63c6ea74-7fa4-4b22-9d03-4c1b33efce39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"
Nov 03 19:58:10.391215 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.391097    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6cbc\" (UniqueName: \"kubernetes.io/projected/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-kube-api-access-v6cbc\") pod \"maas-api-revoke-sub-3d4156f1-qgrf6\" (UID: \"63c6ea74-7fa4-4b22-9d03-4c1b33efce39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"
Nov 03 19:58:10.391722 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.391695    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qgrf6\" (UID: \"63c6ea74-7fa4-4b22-9d03-4c1b33efce39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"
Nov 03 19:58:10.393360 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.393336    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qgrf6\" (UID: \"63c6ea74-7fa4-4b22-9d03-4c1b33efce39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"
Nov 03 19:58:10.403761 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.403732    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6cbc\" (UniqueName: \"kubernetes.io/projected/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-kube-api-access-v6cbc\") pod \"maas-api-revoke-sub-3d4156f1-qgrf6\" (UID: \"63c6ea74-7fa4-4b22-9d03-4c1b33efce39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"
Nov 03 19:58:10.603616 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.603573    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"
Nov 03 19:58:10.879713 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.879685    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"]
Nov 03 19:58:10.882411 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:58:10.882383    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c6ea74_7fa4_4b22_9d03_4c1b33efce39.slice/crio-81940fad2f0423c734b7aa538ed1c9e91dff5c3dbcda8d315b187463510f7ac4 WatchSource:0}: Error finding container 81940fad2f0423c734b7aa538ed1c9e91dff5c3dbcda8d315b187463510f7ac4: Status 404 returned error can't find the container with id 81940fad2f0423c734b7aa538ed1c9e91dff5c3dbcda8d315b187463510f7ac4
Nov 03 19:58:10.960392 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:10.960357    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6" event={"ID":"63c6ea74-7fa4-4b22-9d03-4c1b33efce39","Type":"ContainerStarted","Data":"81940fad2f0423c734b7aa538ed1c9e91dff5c3dbcda8d315b187463510f7ac4"}
Nov 03 19:58:11.858264 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:11.858218    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edf9411f-0622-424d-8672-c5de171d3365" path="/var/lib/kubelet/pods/edf9411f-0622-424d-8672-c5de171d3365/volumes"
Nov 03 19:58:11.966910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:11.966792    5237 generic.go:356] "Generic (PLEG): container finished" podID="63c6ea74-7fa4-4b22-9d03-4c1b33efce39" containerID="0bfd04c7c08f8052d4926f65dc493aa1f64d2ec6bba2c28aca4dead6d8636adf" exitCode=0
Nov 03 19:58:11.966910 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:11.966876    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6" event={"ID":"63c6ea74-7fa4-4b22-9d03-4c1b33efce39","Type":"ContainerDied","Data":"0bfd04c7c08f8052d4926f65dc493aa1f64d2ec6bba2c28aca4dead6d8636adf"}
Nov 03 19:58:13.098286 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:13.098256    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"
Nov 03 19:58:13.115695 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:13.115669    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-kube-api-access-v6cbc\" (UniqueName: \"kubernetes.io/projected/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-kube-api-access-v6cbc\") pod \"63c6ea74-7fa4-4b22-9d03-4c1b33efce39\" (UID: \"63c6ea74-7fa4-4b22-9d03-4c1b33efce39\") "
Nov 03 19:58:13.115784 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:13.115758    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-maas-api-openshift-ca\") pod \"63c6ea74-7fa4-4b22-9d03-4c1b33efce39\" (UID: \"63c6ea74-7fa4-4b22-9d03-4c1b33efce39\") "
Nov 03 19:58:13.115822 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:13.115785    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-maas-api-xks-ca\") pod \"63c6ea74-7fa4-4b22-9d03-4c1b33efce39\" (UID: \"63c6ea74-7fa4-4b22-9d03-4c1b33efce39\") "
Nov 03 19:58:13.116128 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:13.116106    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-maas-api-openshift-ca" pod "63c6ea74-7fa4-4b22-9d03-4c1b33efce39" (UID: "63c6ea74-7fa4-4b22-9d03-4c1b33efce39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:13.118158 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:13.118123    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-maas-api-xks-ca" pod "63c6ea74-7fa4-4b22-9d03-4c1b33efce39" (UID: "63c6ea74-7fa4-4b22-9d03-4c1b33efce39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:58:13.118310 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:13.118290    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-kube-api-access-v6cbc" pod "63c6ea74-7fa4-4b22-9d03-4c1b33efce39" (UID: "63c6ea74-7fa4-4b22-9d03-4c1b33efce39"). InnerVolumeSpecName "kube-api-access-v6cbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:13.216604 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:13.216565    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6cbc\" (UniqueName: \"kubernetes.io/projected/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-kube-api-access-v6cbc\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:13.216756 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:13.216614    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-maas-api-openshift-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:13.216756 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:13.216629    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c6ea74-7fa4-4b22-9d03-4c1b33efce39-maas-api-xks-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:13.255717 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:58:13.255637    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:58:13.977894 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:13.977864    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"
Nov 03 19:58:13.977894 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:13.977875    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6" event={"ID":"63c6ea74-7fa4-4b22-9d03-4c1b33efce39","Type":"ContainerDied","Data":"81940fad2f0423c734b7aa538ed1c9e91dff5c3dbcda8d315b187463510f7ac4"}
Nov 03 19:58:13.978112 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:13.977928    5237 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81940fad2f0423c734b7aa538ed1c9e91dff5c3dbcda8d315b187463510f7ac4"
Nov 03 19:58:31.138549 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:31.138511    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:31.314587 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:31.314550    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:35.762676 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.762643    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"]
Nov 03 19:58:35.763083 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.763023    5237 cpu_manager.go:414] "containerMap: removing container" podUID="63c6ea74-7fa4-4b22-9d03-4c1b33efce39" containerName="revoke-keys"
Nov 03 19:58:35.763083 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.763036    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c6ea74-7fa4-4b22-9d03-4c1b33efce39" containerName="revoke-keys"
Nov 03 19:58:35.763157 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.763102    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c6ea74-7fa4-4b22-9d03-4c1b33efce39" containerName="revoke-keys"
Nov 03 19:58:35.766303 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.766285    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"
Nov 03 19:58:35.773446 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.772971    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:58:35.774332 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.774301    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:58:35.776765 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.776743    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"]
Nov 03 19:58:35.779865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.779819    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:58:35.779865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.779824    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\""
Nov 03 19:58:35.863144 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.863110    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px64k\" (UniqueName: \"kubernetes.io/projected/5843167c-f9fb-4179-b2d4-e43bae846bbc-kube-api-access-px64k\") pod \"maas-api-revoke-sub-e2fc9c5b-9wj5x\" (UID: \"5843167c-f9fb-4179-b2d4-e43bae846bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"
Nov 03 19:58:35.863347 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.863189    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5843167c-f9fb-4179-b2d4-e43bae846bbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9wj5x\" (UID: \"5843167c-f9fb-4179-b2d4-e43bae846bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"
Nov 03 19:58:35.863347 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.863220    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5843167c-f9fb-4179-b2d4-e43bae846bbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9wj5x\" (UID: \"5843167c-f9fb-4179-b2d4-e43bae846bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"
Nov 03 19:58:35.964467 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.964427    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px64k\" (UniqueName: \"kubernetes.io/projected/5843167c-f9fb-4179-b2d4-e43bae846bbc-kube-api-access-px64k\") pod \"maas-api-revoke-sub-e2fc9c5b-9wj5x\" (UID: \"5843167c-f9fb-4179-b2d4-e43bae846bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"
Nov 03 19:58:35.964680 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.964491    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5843167c-f9fb-4179-b2d4-e43bae846bbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9wj5x\" (UID: \"5843167c-f9fb-4179-b2d4-e43bae846bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"
Nov 03 19:58:35.964680 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.964512    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5843167c-f9fb-4179-b2d4-e43bae846bbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9wj5x\" (UID: \"5843167c-f9fb-4179-b2d4-e43bae846bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"
Nov 03 19:58:35.965268 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.965243    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5843167c-f9fb-4179-b2d4-e43bae846bbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9wj5x\" (UID: \"5843167c-f9fb-4179-b2d4-e43bae846bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"
Nov 03 19:58:35.966938 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.966915    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5843167c-f9fb-4179-b2d4-e43bae846bbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9wj5x\" (UID: \"5843167c-f9fb-4179-b2d4-e43bae846bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"
Nov 03 19:58:35.977040 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:35.977016    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px64k\" (UniqueName: \"kubernetes.io/projected/5843167c-f9fb-4179-b2d4-e43bae846bbc-kube-api-access-px64k\") pod \"maas-api-revoke-sub-e2fc9c5b-9wj5x\" (UID: \"5843167c-f9fb-4179-b2d4-e43bae846bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"
Nov 03 19:58:36.080435 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:36.080400    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"
Nov 03 19:58:36.354147 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:36.354119    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"]
Nov 03 19:58:36.356529 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:58:36.356503    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5843167c_f9fb_4179_b2d4_e43bae846bbc.slice/crio-1e47dfbb88bc82d3f740850a4d1864cd2d57a736912506142b2409b8ac1b503d WatchSource:0}: Error finding container 1e47dfbb88bc82d3f740850a4d1864cd2d57a736912506142b2409b8ac1b503d: Status 404 returned error can't find the container with id 1e47dfbb88bc82d3f740850a4d1864cd2d57a736912506142b2409b8ac1b503d
Nov 03 19:58:37.086431 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:37.086390    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x" event={"ID":"5843167c-f9fb-4179-b2d4-e43bae846bbc","Type":"ContainerStarted","Data":"4898c1d88826d514951b3870f7d47d67e74ab880a5ac23f2a47e19df5a6a3676"}
Nov 03 19:58:37.086854 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:37.086487    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x" event={"ID":"5843167c-f9fb-4179-b2d4-e43bae846bbc","Type":"ContainerStarted","Data":"1e47dfbb88bc82d3f740850a4d1864cd2d57a736912506142b2409b8ac1b503d"}
Nov 03 19:58:37.107665 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:37.107609    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x" podStartSLOduration=2.107596835 podStartE2EDuration="2.107596835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:37.104911505 +0000 UTC m=+1873.359218730" watchObservedRunningTime="2026-11-03 19:58:37.107596835 +0000 UTC m=+1873.361904103"
Nov 03 19:58:38.092012 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:38.091979    5237 generic.go:356] "Generic (PLEG): container finished" podID="5843167c-f9fb-4179-b2d4-e43bae846bbc" containerID="4898c1d88826d514951b3870f7d47d67e74ab880a5ac23f2a47e19df5a6a3676" exitCode=0
Nov 03 19:58:38.092415 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:38.092047    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x" event={"ID":"5843167c-f9fb-4179-b2d4-e43bae846bbc","Type":"ContainerDied","Data":"4898c1d88826d514951b3870f7d47d67e74ab880a5ac23f2a47e19df5a6a3676"}
Nov 03 19:58:39.224342 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:39.224315    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"
Nov 03 19:58:39.397017 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:39.396917    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5843167c-f9fb-4179-b2d4-e43bae846bbc-kube-api-access-px64k\" (UniqueName: \"kubernetes.io/projected/5843167c-f9fb-4179-b2d4-e43bae846bbc-kube-api-access-px64k\") pod \"5843167c-f9fb-4179-b2d4-e43bae846bbc\" (UID: \"5843167c-f9fb-4179-b2d4-e43bae846bbc\") "
Nov 03 19:58:39.397017 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:39.396965    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5843167c-f9fb-4179-b2d4-e43bae846bbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5843167c-f9fb-4179-b2d4-e43bae846bbc-maas-api-openshift-ca\") pod \"5843167c-f9fb-4179-b2d4-e43bae846bbc\" (UID: \"5843167c-f9fb-4179-b2d4-e43bae846bbc\") "
Nov 03 19:58:39.397017 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:39.397014    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5843167c-f9fb-4179-b2d4-e43bae846bbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5843167c-f9fb-4179-b2d4-e43bae846bbc-maas-api-xks-ca\") pod \"5843167c-f9fb-4179-b2d4-e43bae846bbc\" (UID: \"5843167c-f9fb-4179-b2d4-e43bae846bbc\") "
Nov 03 19:58:39.397365 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:39.397341    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5843167c-f9fb-4179-b2d4-e43bae846bbc-maas-api-openshift-ca" pod "5843167c-f9fb-4179-b2d4-e43bae846bbc" (UID: "5843167c-f9fb-4179-b2d4-e43bae846bbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:39.399200 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:39.399176    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5843167c-f9fb-4179-b2d4-e43bae846bbc-kube-api-access-px64k" pod "5843167c-f9fb-4179-b2d4-e43bae846bbc" (UID: "5843167c-f9fb-4179-b2d4-e43bae846bbc"). InnerVolumeSpecName "kube-api-access-px64k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:39.399200 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:39.399180    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5843167c-f9fb-4179-b2d4-e43bae846bbc-maas-api-xks-ca" pod "5843167c-f9fb-4179-b2d4-e43bae846bbc" (UID: "5843167c-f9fb-4179-b2d4-e43bae846bbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:58:39.498559 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:39.498517    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px64k\" (UniqueName: \"kubernetes.io/projected/5843167c-f9fb-4179-b2d4-e43bae846bbc-kube-api-access-px64k\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:39.498559 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:39.498552    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5843167c-f9fb-4179-b2d4-e43bae846bbc-maas-api-openshift-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:39.498559 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:39.498564    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5843167c-f9fb-4179-b2d4-e43bae846bbc-maas-api-xks-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:40.103021 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:40.102991    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"
Nov 03 19:58:40.103021 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:40.103011    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x" event={"ID":"5843167c-f9fb-4179-b2d4-e43bae846bbc","Type":"ContainerDied","Data":"1e47dfbb88bc82d3f740850a4d1864cd2d57a736912506142b2409b8ac1b503d"}
Nov 03 19:58:40.103243 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:40.103068    5237 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e47dfbb88bc82d3f740850a4d1864cd2d57a736912506142b2409b8ac1b503d"
Nov 03 19:58:43.258540 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:58:43.258499    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:58:48.993002 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:48.992962    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:49.139004 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:58:49.138966    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:02.055409 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:02.055371    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"]
Nov 03 19:59:02.060104 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:02.060070    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgrf6"]
Nov 03 19:59:03.858682 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:03.858645    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c6ea74-7fa4-4b22-9d03-4c1b33efce39" path="/var/lib/kubelet/pods/63c6ea74-7fa4-4b22-9d03-4c1b33efce39/volumes"
Nov 03 19:59:13.255324 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:59:13.255288    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:59:17.244854 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:17.244807    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:17.727156 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:17.727105    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:17.884987 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:17.884945    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:21.600364 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:21.600322    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:21.757129 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:21.757076    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:22.113457 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:22.113418    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"]
Nov 03 19:59:22.125134 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:22.125103    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q9r7z"]
Nov 03 19:59:22.961864 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:22.961812    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:23.181857 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:23.178424    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:23.391195 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:23.391153    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:23.858100 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:23.858061    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fed683d-6d7c-4a17-895a-c3689b7ec02d" path="/var/lib/kubelet/pods/5fed683d-6d7c-4a17-895a-c3689b7ec02d/volumes"
Nov 03 19:59:23.988449 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:23.988407    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:24.881103 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:24.881070    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:25.674496 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:25.674458    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:26.545150 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:26.545106    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:27.134250 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:27.134196    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:27.853079 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:27.852986    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:28.514433 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:28.514392    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:29.118493 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:29.118452    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:29.849883 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:29.849818    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:30.530308 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:30.530272    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:31.144039 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:31.140744    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:31.271389 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:31.271348    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:31.844401 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:31.844340    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:32.431703 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:32.431664    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:32.574550 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:32.574512    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:32.732695 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:32.732609    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:33.433482 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:33.433442    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:34.047130 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:34.047087    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:34.806688 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:34.806648    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:35.418244 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:35.418209    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:36.101461 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:36.101425    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:36.635819 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:36.635777    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:36.775710 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:36.775678    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:37.253210 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:37.253172    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:37.398162 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:37.398124    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:37.577480 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:37.577440    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:38.029809 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.029719    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"]
Nov 03 19:59:38.030288 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.030254    5237 cpu_manager.go:414] "containerMap: removing container" podUID="5843167c-f9fb-4179-b2d4-e43bae846bbc" containerName="revoke-keys"
Nov 03 19:59:38.030288 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.030287    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5843167c-f9fb-4179-b2d4-e43bae846bbc" containerName="revoke-keys"
Nov 03 19:59:38.030469 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.030386    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="5843167c-f9fb-4179-b2d4-e43bae846bbc" containerName="revoke-keys"
Nov 03 19:59:38.033802 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.033783    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.039779 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.039757    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-458393\"/\"default-dockercfg-xgh42\""
Nov 03 19:59:38.039779 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.039757    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-458393\"/\"simulator-single-main-458393-kserve-self-signed-certs\""
Nov 03 19:59:38.040145 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.040123    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-458393\"/\"openshift-service-ca.crt\""
Nov 03 19:59:38.040287 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.040132    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-458393\"/\"kube-root-ca.crt\""
Nov 03 19:59:38.061872 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.061820    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"]
Nov 03 19:59:38.074555 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.074522    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-tmp-dir\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.074729 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.074565    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-model-cache\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.074729 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.074698    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grnbd\" (UniqueName: \"kubernetes.io/projected/b143984d-dd0c-4577-ae76-67de0cf11aad-kube-api-access-grnbd\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.074889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.074779    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b143984d-dd0c-4577-ae76-67de0cf11aad-tls-certs\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.074889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.074801    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-home\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.074889 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.074829    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-dshm\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.149546 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.149509    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:38.175476 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.175439    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-tmp-dir\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.175637 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.175483    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-model-cache\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.175637 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.175512    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grnbd\" (UniqueName: \"kubernetes.io/projected/b143984d-dd0c-4577-ae76-67de0cf11aad-kube-api-access-grnbd\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.175637 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.175529    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b143984d-dd0c-4577-ae76-67de0cf11aad-tls-certs\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.175637 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.175547    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-home\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.175637 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.175569    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-dshm\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.175939 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.175919    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-tmp-dir\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.175987 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.175957    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-model-cache\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.176030 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.175985    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-home\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.177797 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.177773    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-dshm\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.178140 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.178122    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b143984d-dd0c-4577-ae76-67de0cf11aad-tls-certs\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.191424 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.191401    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grnbd\" (UniqueName: \"kubernetes.io/projected/b143984d-dd0c-4577-ae76-67de0cf11aad-kube-api-access-grnbd\") pod \"simulator-single-main-458393-kserve-78487c46fb-mlmh6\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") " pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.346526 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.346486    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:38.577859 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.577225    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:38.632624 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:38.632533    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"]
Nov 03 19:59:38.639624 ip-10-0-133-29 kubenswrapper[5237]: W1103 19:59:38.639594    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb143984d_dd0c_4577_ae76_67de0cf11aad.slice/crio-13f947a5055017f963078c919a60760756b01e7cdf88fe6e2e470351b696692f WatchSource:0}: Error finding container 13f947a5055017f963078c919a60760756b01e7cdf88fe6e2e470351b696692f: Status 404 returned error can't find the container with id 13f947a5055017f963078c919a60760756b01e7cdf88fe6e2e470351b696692f
Nov 03 19:59:39.395517 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:39.395482    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6" event={"ID":"b143984d-dd0c-4577-ae76-67de0cf11aad","Type":"ContainerStarted","Data":"6408594f7b90601943ea63a5d35b640e9216bc10a19b842ba04a95af89a5a2c6"}
Nov 03 19:59:39.395961 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:39.395530    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6" event={"ID":"b143984d-dd0c-4577-ae76-67de0cf11aad","Type":"ContainerStarted","Data":"13f947a5055017f963078c919a60760756b01e7cdf88fe6e2e470351b696692f"}
Nov 03 19:59:39.417125 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:39.417075    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6" podStartSLOduration=1.417059307 podStartE2EDuration="1.417059307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:39.4156883 +0000 UTC m=+1935.669995527" watchObservedRunningTime="2026-11-03 19:59:39.417059307 +0000 UTC m=+1935.671366532"
Nov 03 19:59:39.432053 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:39.432000    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:40.347702 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:40.347664    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:40.378129 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:40.376856    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:41.357324 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:41.357254    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:42.035168 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:42.034915    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:42.732410 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:42.732367    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:43.255542 ip-10-0-133-29 kubenswrapper[5237]: E1103 19:59:43.255506    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 19:59:43.333896 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:43.333858    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:43.945865 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:43.945818    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:44.618757 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:44.618721    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:45.241136 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:45.241096    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:45.855075 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:45.855034    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:46.528896 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:46.528854    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:47.154408 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:47.154372    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:47.814350 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:47.814317    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:48.347063 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:48.347022    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:48.359780 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:48.359747    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:48.432717 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:48.432678    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:48.462507 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:48.462478    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 19:59:49.220184 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:49.220145    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:50.004816 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:50.004775    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:50.746930 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:50.746175    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:51.413395 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:51.413361    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:52.068600 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:52.068552    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:52.842916 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:52.842827    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:53.590974 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:53.590935    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:54.125005 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:54.124962    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:54.933182 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:54.933148    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:55.568862 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:55.568296    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:56.386299 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:56.386264    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:57.053614 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:57.053135    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:57.812877 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:57.812806    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:58.424312 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:58.424274    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:58.573018 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:58.572981    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:59.269443 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:59.269396    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:59.932333 ip-10-0-133-29 kubenswrapper[5237]: I1103 19:59:59.932293    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:00.152384 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:00.152352    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74"]
Nov 03 20:00:00.156087 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:00.156063    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74"
Nov 03 20:00:00.162674 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:00.162650    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\""
Nov 03 20:00:00.164148 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:00.164126    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:00:00.164310 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:00.164242    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:00:00.171794 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:00.171769    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74"]
Nov 03 20:00:00.207245 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:00.207163    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff7e2c7-3ae9-4c64-8769-fec73729b1d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74\" (UID: \"1ff7e2c7-3ae9-4c64-8769-fec73729b1d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74"
Nov 03 20:00:00.207406 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:00.207292    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4blbk\" (UniqueName: \"kubernetes.io/projected/1ff7e2c7-3ae9-4c64-8769-fec73729b1d0-kube-api-access-4blbk\") pod \"maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74\" (UID: \"1ff7e2c7-3ae9-4c64-8769-fec73729b1d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74"
Nov 03 20:00:00.308689 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:00.308584    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4blbk\" (UniqueName: \"kubernetes.io/projected/1ff7e2c7-3ae9-4c64-8769-fec73729b1d0-kube-api-access-4blbk\") pod \"maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74\" (UID: \"1ff7e2c7-3ae9-4c64-8769-fec73729b1d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74"
Nov 03 20:00:00.309294 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:00.308749    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff7e2c7-3ae9-4c64-8769-fec73729b1d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74\" (UID: \"1ff7e2c7-3ae9-4c64-8769-fec73729b1d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74"
Nov 03 20:00:00.309622 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:00.309596    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff7e2c7-3ae9-4c64-8769-fec73729b1d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74\" (UID: \"1ff7e2c7-3ae9-4c64-8769-fec73729b1d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74"
Nov 03 20:00:00.323304 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:00.323277    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4blbk\" (UniqueName: \"kubernetes.io/projected/1ff7e2c7-3ae9-4c64-8769-fec73729b1d0-kube-api-access-4blbk\") pod \"maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74\" (UID: \"1ff7e2c7-3ae9-4c64-8769-fec73729b1d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74"
Nov 03 20:00:00.470499 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:00.470418    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74"
Nov 03 20:00:00.641649 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:00.641613    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:00.751375 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:00.751347    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74"]
Nov 03 20:00:01.440611 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:01.440458    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:01.511201 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:01.511167    5237 generic.go:356] "Generic (PLEG): container finished" podID="1ff7e2c7-3ae9-4c64-8769-fec73729b1d0" containerID="d46ea8384df5f1eea1612a4a95b0aa682319914935c4da4a74cfd91246fcb453" exitCode=60
Nov 03 20:00:01.511409 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:01.511235    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74" event={"ID":"1ff7e2c7-3ae9-4c64-8769-fec73729b1d0","Type":"ContainerDied","Data":"d46ea8384df5f1eea1612a4a95b0aa682319914935c4da4a74cfd91246fcb453"}
Nov 03 20:00:01.511409 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:01.511291    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74" event={"ID":"1ff7e2c7-3ae9-4c64-8769-fec73729b1d0","Type":"ContainerStarted","Data":"d8f5725679305a5973d583fd748a640288b7da62e536c2aa87c02e2f3023dd08"}
Nov 03 20:00:01.511572 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:01.511555    5237 scope.go:122] "RemoveContainer" containerID="d46ea8384df5f1eea1612a4a95b0aa682319914935c4da4a74cfd91246fcb453"
Nov 03 20:00:01.784447 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:01.784417    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:01.924081 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:01.924051    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:02.225172 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:02.225130    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:02.518260 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:02.518170    5237 generic.go:356] "Generic (PLEG): container finished" podID="1ff7e2c7-3ae9-4c64-8769-fec73729b1d0" containerID="32ec868f3d56373b5f7a780927050b7a5534cf8fcbc5d9a222206d9f77f6c770" exitCode=60
Nov 03 20:00:02.518743 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:02.518261    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74" event={"ID":"1ff7e2c7-3ae9-4c64-8769-fec73729b1d0","Type":"ContainerDied","Data":"32ec868f3d56373b5f7a780927050b7a5534cf8fcbc5d9a222206d9f77f6c770"}
Nov 03 20:00:02.518743 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:02.518320    5237 scope.go:122] "RemoveContainer" containerID="d46ea8384df5f1eea1612a4a95b0aa682319914935c4da4a74cfd91246fcb453"
Nov 03 20:00:02.518743 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:02.518603    5237 scope.go:122] "RemoveContainer" containerID="32ec868f3d56373b5f7a780927050b7a5534cf8fcbc5d9a222206d9f77f6c770"
Nov 03 20:00:02.518938 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:00:02.518828    5237 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74_odh-ai-gateway-infra(1ff7e2c7-3ae9-4c64-8769-fec73729b1d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74" podUID="1ff7e2c7-3ae9-4c64-8769-fec73729b1d0"
Nov 03 20:00:02.904336 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:02.904303    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:03.524985 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:03.524959    5237 scope.go:122] "RemoveContainer" containerID="32ec868f3d56373b5f7a780927050b7a5534cf8fcbc5d9a222206d9f77f6c770"
Nov 03 20:00:03.525413 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:00:03.525175    5237 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74_odh-ai-gateway-infra(1ff7e2c7-3ae9-4c64-8769-fec73729b1d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74" podUID="1ff7e2c7-3ae9-4c64-8769-fec73729b1d0"
Nov 03 20:00:03.638503 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:03.638463    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:04.311612 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:04.311569    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:04.939091 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:04.939052    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:05.658639 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:05.658595    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:06.558856 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:06.558793    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:07.254042 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:07.253989    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:08.023543 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:08.023502    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:08.682263 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:08.682223    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:09.314972 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:09.314927    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:09.969137 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:09.969101    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:10.675304 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:10.675263    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:11.455860 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:11.446380    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:11.613190 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:11.613094    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:12.162431 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:12.162366    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:12.958338 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:12.958299    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:13.196714 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:13.196678    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:13.257512 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:00:13.257427    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:00:13.399945 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:13.399916    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:13.764515 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:13.764478    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:13.855667 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:13.855640    5237 scope.go:122] "RemoveContainer" containerID="32ec868f3d56373b5f7a780927050b7a5534cf8fcbc5d9a222206d9f77f6c770"
Nov 03 20:00:14.459498 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:14.459397    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:14.602216 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:14.602181    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74" event={"ID":"1ff7e2c7-3ae9-4c64-8769-fec73729b1d0","Type":"ContainerStarted","Data":"5bced8e58639e113fab38dd34e654b12b98ee50fa098b0cade5f1245589e81f1"}
Nov 03 20:00:14.622469 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:14.622417    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74" podStartSLOduration=14.622402667 podStartE2EDuration="14.622402667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:14.62039125 +0000 UTC m=+1970.874698489" watchObservedRunningTime="2026-11-03 20:00:14.622402667 +0000 UTC m=+1970.876709892"
Nov 03 20:00:15.126943 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:15.126875    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:15.609192 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:15.609157    5237 generic.go:356] "Generic (PLEG): container finished" podID="1ff7e2c7-3ae9-4c64-8769-fec73729b1d0" containerID="5bced8e58639e113fab38dd34e654b12b98ee50fa098b0cade5f1245589e81f1" exitCode=60
Nov 03 20:00:15.609664 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:15.609213    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74" event={"ID":"1ff7e2c7-3ae9-4c64-8769-fec73729b1d0","Type":"ContainerDied","Data":"5bced8e58639e113fab38dd34e654b12b98ee50fa098b0cade5f1245589e81f1"}
Nov 03 20:00:15.609664 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:15.609269    5237 scope.go:122] "RemoveContainer" containerID="32ec868f3d56373b5f7a780927050b7a5534cf8fcbc5d9a222206d9f77f6c770"
Nov 03 20:00:15.609664 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:15.609641    5237 scope.go:122] "RemoveContainer" containerID="5bced8e58639e113fab38dd34e654b12b98ee50fa098b0cade5f1245589e81f1"
Nov 03 20:00:15.609936 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:00:15.609909    5237 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74_odh-ai-gateway-infra(1ff7e2c7-3ae9-4c64-8769-fec73729b1d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74" podUID="1ff7e2c7-3ae9-4c64-8769-fec73729b1d0"
Nov 03 20:00:15.640452 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:15.640418    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74"]
Nov 03 20:00:15.780298 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:15.780234    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:16.729468 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:16.729435    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:16.768706 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:16.768681    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74"
Nov 03 20:00:16.876809 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:16.876714    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ff7e2c7-3ae9-4c64-8769-fec73729b1d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff7e2c7-3ae9-4c64-8769-fec73729b1d0-maas-api-service-ca\") pod \"1ff7e2c7-3ae9-4c64-8769-fec73729b1d0\" (UID: \"1ff7e2c7-3ae9-4c64-8769-fec73729b1d0\") "
Nov 03 20:00:16.877004 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:16.876888    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff7e2c7-3ae9-4c64-8769-fec73729b1d0-kube-api-access-4blbk\" (UniqueName: \"kubernetes.io/projected/1ff7e2c7-3ae9-4c64-8769-fec73729b1d0-kube-api-access-4blbk\") pod \"1ff7e2c7-3ae9-4c64-8769-fec73729b1d0\" (UID: \"1ff7e2c7-3ae9-4c64-8769-fec73729b1d0\") "
Nov 03 20:00:16.877139 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:16.877117    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff7e2c7-3ae9-4c64-8769-fec73729b1d0-maas-api-service-ca" pod "1ff7e2c7-3ae9-4c64-8769-fec73729b1d0" (UID: "1ff7e2c7-3ae9-4c64-8769-fec73729b1d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:16.879027 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:16.879002    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff7e2c7-3ae9-4c64-8769-fec73729b1d0-kube-api-access-4blbk" pod "1ff7e2c7-3ae9-4c64-8769-fec73729b1d0" (UID: "1ff7e2c7-3ae9-4c64-8769-fec73729b1d0"). InnerVolumeSpecName "kube-api-access-4blbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:16.977588 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:16.977547    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4blbk\" (UniqueName: \"kubernetes.io/projected/1ff7e2c7-3ae9-4c64-8769-fec73729b1d0-kube-api-access-4blbk\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:16.977588 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:16.977581    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff7e2c7-3ae9-4c64-8769-fec73729b1d0-maas-api-service-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:17.467144 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:17.467109    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:17.622279 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:17.622242    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74"
Nov 03 20:00:17.622279 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:17.622269    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74" event={"ID":"1ff7e2c7-3ae9-4c64-8769-fec73729b1d0","Type":"ContainerDied","Data":"d8f5725679305a5973d583fd748a640288b7da62e536c2aa87c02e2f3023dd08"}
Nov 03 20:00:17.622549 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:17.622346    5237 scope.go:122] "RemoveContainer" containerID="5bced8e58639e113fab38dd34e654b12b98ee50fa098b0cade5f1245589e81f1"
Nov 03 20:00:17.646005 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:17.645972    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74"]
Nov 03 20:00:17.653577 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:17.653549    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-458393-2989560smv74"]
Nov 03 20:00:17.858282 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:17.858248    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff7e2c7-3ae9-4c64-8769-fec73729b1d0" path="/var/lib/kubelet/pods/1ff7e2c7-3ae9-4c64-8769-fec73729b1d0/volumes"
Nov 03 20:00:18.235175 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:18.235029    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:18.956729 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:18.956692    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:19.737436 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:19.737354    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:19.886266 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:19.886227    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:20.025514 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:20.025420    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"]
Nov 03 20:00:20.027412 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:20.027389    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vr2ph"]
Nov 03 20:00:20.544568 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:20.544528    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:21.463562 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:21.463523    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:21.857943 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:21.857907    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59cbdbc9-2495-4374-b013-12777a2bc983" path="/var/lib/kubelet/pods/59cbdbc9-2495-4374-b013-12777a2bc983/volumes"
Nov 03 20:00:22.153884 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:22.149490    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:22.860300 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:22.855123    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:23.548495 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:23.548452    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:24.261347 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:24.260765    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:24.692369 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:24.692332    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"]
Nov 03 20:00:24.814545 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:24.814508    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:24.977278 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:24.977173    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:25.283939 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:25.283825    5237 scope.go:122] "RemoveContainer" containerID="ba41c00da425fa52a1f962fd12f2a9d52d14314992c9667eb3cf9de0c771045c"
Nov 03 20:00:25.556588 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:25.556476    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:26.251426 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:26.251353    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:27.333598 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:27.333563    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:27.882895 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:27.882855    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:28.731883 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:28.731793    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:29.432757 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:29.432721    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:30.041774 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:30.041733    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:30.177579 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:30.177523    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:30.451282 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:30.451167    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:30.624338 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:30.624294    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:30.989795 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:30.989759    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:31.260870 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:31.260756    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:36.908135 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:36.908039    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:37.153671 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:37.153623    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:37.313920 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:37.313883    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:39.023660 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:39.023625    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"]
Nov 03 20:00:39.026371 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:39.026342    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bw8wj"]
Nov 03 20:00:39.712775 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:39.712732    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6" podUID="b143984d-dd0c-4577-ae76-67de0cf11aad" containerName="main" containerID="cri-o://6408594f7b90601943ea63a5d35b640e9216bc10a19b842ba04a95af89a5a2c6" gracePeriod=45
Nov 03 20:00:39.858890 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:39.858859    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6b00742-24ad-45a6-a535-b0100043b95c" path="/var/lib/kubelet/pods/c6b00742-24ad-45a6-a535-b0100043b95c/volumes"
Nov 03 20:00:39.960171 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:39.960146    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 20:00:40.002548 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.002454    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b143984d-dd0c-4577-ae76-67de0cf11aad-kube-api-access-grnbd\" (UniqueName: \"kubernetes.io/projected/b143984d-dd0c-4577-ae76-67de0cf11aad-kube-api-access-grnbd\") pod \"b143984d-dd0c-4577-ae76-67de0cf11aad\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") "
Nov 03 20:00:40.002548 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.002509    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b143984d-dd0c-4577-ae76-67de0cf11aad-tls-certs\" (UniqueName: \"kubernetes.io/secret/b143984d-dd0c-4577-ae76-67de0cf11aad-tls-certs\") pod \"b143984d-dd0c-4577-ae76-67de0cf11aad\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") "
Nov 03 20:00:40.002785 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.002560    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-tmp-dir\") pod \"b143984d-dd0c-4577-ae76-67de0cf11aad\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") "
Nov 03 20:00:40.002785 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.002594    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-dshm\") pod \"b143984d-dd0c-4577-ae76-67de0cf11aad\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") "
Nov 03 20:00:40.002785 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.002628    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-home\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-home\") pod \"b143984d-dd0c-4577-ae76-67de0cf11aad\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") "
Nov 03 20:00:40.002785 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.002683    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-model-cache\") pod \"b143984d-dd0c-4577-ae76-67de0cf11aad\" (UID: \"b143984d-dd0c-4577-ae76-67de0cf11aad\") "
Nov 03 20:00:40.003053 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.002919    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-tmp-dir" pod "b143984d-dd0c-4577-ae76-67de0cf11aad" (UID: "b143984d-dd0c-4577-ae76-67de0cf11aad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:40.003209 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.003157    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-model-cache" pod "b143984d-dd0c-4577-ae76-67de0cf11aad" (UID: "b143984d-dd0c-4577-ae76-67de0cf11aad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:40.003209 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.003165    5237 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-tmp-dir\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:40.003485 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.003452    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-home" pod "b143984d-dd0c-4577-ae76-67de0cf11aad" (UID: "b143984d-dd0c-4577-ae76-67de0cf11aad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:40.005095 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.005069    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b143984d-dd0c-4577-ae76-67de0cf11aad-tls-certs" pod "b143984d-dd0c-4577-ae76-67de0cf11aad" (UID: "b143984d-dd0c-4577-ae76-67de0cf11aad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:00:40.005215 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.005126    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b143984d-dd0c-4577-ae76-67de0cf11aad-kube-api-access-grnbd" pod "b143984d-dd0c-4577-ae76-67de0cf11aad" (UID: "b143984d-dd0c-4577-ae76-67de0cf11aad"). InnerVolumeSpecName "kube-api-access-grnbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:40.006646 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.006624    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-dshm" pod "b143984d-dd0c-4577-ae76-67de0cf11aad" (UID: "b143984d-dd0c-4577-ae76-67de0cf11aad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:40.104447 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.104403    5237 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-dshm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:40.104447 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.104439    5237 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-home\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:40.104447 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.104454    5237 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b143984d-dd0c-4577-ae76-67de0cf11aad-model-cache\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:40.104955 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.104470    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grnbd\" (UniqueName: \"kubernetes.io/projected/b143984d-dd0c-4577-ae76-67de0cf11aad-kube-api-access-grnbd\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:40.104955 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.104484    5237 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b143984d-dd0c-4577-ae76-67de0cf11aad-tls-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:40.744949 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.744907    5237 generic.go:356] "Generic (PLEG): container finished" podID="b143984d-dd0c-4577-ae76-67de0cf11aad" containerID="6408594f7b90601943ea63a5d35b640e9216bc10a19b842ba04a95af89a5a2c6" exitCode=0
Nov 03 20:00:40.745150 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.744967    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"
Nov 03 20:00:40.745150 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.744998    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6" event={"ID":"b143984d-dd0c-4577-ae76-67de0cf11aad","Type":"ContainerDied","Data":"6408594f7b90601943ea63a5d35b640e9216bc10a19b842ba04a95af89a5a2c6"}
Nov 03 20:00:40.745150 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.745047    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6" event={"ID":"b143984d-dd0c-4577-ae76-67de0cf11aad","Type":"ContainerDied","Data":"13f947a5055017f963078c919a60760756b01e7cdf88fe6e2e470351b696692f"}
Nov 03 20:00:40.745150 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.745066    5237 scope.go:122] "RemoveContainer" containerID="6408594f7b90601943ea63a5d35b640e9216bc10a19b842ba04a95af89a5a2c6"
Nov 03 20:00:40.754683 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.754660    5237 scope.go:122] "RemoveContainer" containerID="6408594f7b90601943ea63a5d35b640e9216bc10a19b842ba04a95af89a5a2c6"
Nov 03 20:00:40.755012 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:00:40.754991    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6408594f7b90601943ea63a5d35b640e9216bc10a19b842ba04a95af89a5a2c6\": container with ID starting with 6408594f7b90601943ea63a5d35b640e9216bc10a19b842ba04a95af89a5a2c6 not found: ID does not exist" containerID="6408594f7b90601943ea63a5d35b640e9216bc10a19b842ba04a95af89a5a2c6"
Nov 03 20:00:40.755084 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.755020    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6408594f7b90601943ea63a5d35b640e9216bc10a19b842ba04a95af89a5a2c6"} err="failed to get container status \"6408594f7b90601943ea63a5d35b640e9216bc10a19b842ba04a95af89a5a2c6\": rpc error: code = NotFound desc = could not find container \"6408594f7b90601943ea63a5d35b640e9216bc10a19b842ba04a95af89a5a2c6\": container with ID starting with 6408594f7b90601943ea63a5d35b640e9216bc10a19b842ba04a95af89a5a2c6 not found: ID does not exist"
Nov 03 20:00:40.772217 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.772189    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"]
Nov 03 20:00:40.781358 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:40.781327    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-458393/simulator-single-main-458393-kserve-78487c46fb-mlmh6"]
Nov 03 20:00:41.858113 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:41.858077    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b143984d-dd0c-4577-ae76-67de0cf11aad" path="/var/lib/kubelet/pods/b143984d-dd0c-4577-ae76-67de0cf11aad/volumes"
Nov 03 20:00:43.254809 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:00:43.254770    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:00:49.023609 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:49.023575    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"]
Nov 03 20:00:49.028013 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:49.027977    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dhnfj"]
Nov 03 20:00:49.858196 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:00:49.858150    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da6f60e0-ab2b-4c15-8030-995c1a7a243d" path="/var/lib/kubelet/pods/da6f60e0-ab2b-4c15-8030-995c1a7a243d/volumes"
Nov 03 20:01:02.027023 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:01:02.026965    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:02.402307 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:01:02.402266    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:04.841145 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:01:04.841094    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:08.202512 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:01:08.202461    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:13.255250 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:01:13.255214    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:01:16.032253 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:01:16.032215    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"]
Nov 03 20:01:16.038102 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:01:16.038077    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bg6ps"]
Nov 03 20:01:17.857782 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:01:17.857748    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6336a7a-38a0-4d29-855e-192cf3b737f7" path="/var/lib/kubelet/pods/f6336a7a-38a0-4d29-855e-192cf3b737f7/volumes"
Nov 03 20:01:25.310223 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:01:25.310186    5237 scope.go:122] "RemoveContainer" containerID="750133c4bf10bfe68380bb8473af910c3d9c22f9aad18cc7d1c5f108d88f4376"
Nov 03 20:01:26.038184 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:01:26.038147    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"]
Nov 03 20:01:26.043082 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:01:26.043054    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vs6ds"]
Nov 03 20:01:27.857938 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:01:27.857897    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b31c84e-3dbd-4dfd-a807-9f9019fa464c" path="/var/lib/kubelet/pods/2b31c84e-3dbd-4dfd-a807-9f9019fa464c/volumes"
Nov 03 20:01:31.427705 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:01:31.427664    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:43.255132 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:01:43.255093    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:01:47.073440 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:01:47.073398    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"]
Nov 03 20:01:47.080484 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:01:47.080453    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqdhh"]
Nov 03 20:01:47.858042 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:01:47.858004    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c79580-7caf-4796-a47d-a01f6a16fb73" path="/var/lib/kubelet/pods/98c79580-7caf-4796-a47d-a01f6a16fb73/volumes"
Nov 03 20:02:05.030072 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:05.029977    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"]
Nov 03 20:02:05.032056 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:05.032030    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-l7fg5"]
Nov 03 20:02:05.841578 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:05.841535    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:05.858763 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:05.858729    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88c4a7ac-0729-400f-99fb-a7bea0bf372b" path="/var/lib/kubelet/pods/88c4a7ac-0729-400f-99fb-a7bea0bf372b/volumes"
Nov 03 20:02:13.256159 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:02:13.256116    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:02:24.022864 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:24.022817    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"]
Nov 03 20:02:24.025622 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:24.025594    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xp2kr"]
Nov 03 20:02:25.333644 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:25.333604    5237 scope.go:122] "RemoveContainer" containerID="0d74c43821f6f44ce7596b4f9b6f8a69a9e0a30d12d636aefbfc6a2e2eb4ab51"
Nov 03 20:02:25.342251 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:25.342230    5237 scope.go:122] "RemoveContainer" containerID="7911cb08ae1bb96239f214b349e30f23867f7dce35bc61bc8dc8bf0e3010ec6b"
Nov 03 20:02:25.349748 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:25.349723    5237 scope.go:122] "RemoveContainer" containerID="c5c8f06a4a3151d37e7fcdebec17725322faa60e1ec98fd30daed300da243d07"
Nov 03 20:02:25.357109 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:25.357087    5237 scope.go:122] "RemoveContainer" containerID="47e91deed890138e907804a3752790b8779a3e4158b0cc8a165e6cb87e976439"
Nov 03 20:02:25.857848 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:25.857803    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30ed0d74-db0f-400a-930c-e889d9776754" path="/var/lib/kubelet/pods/30ed0d74-db0f-400a-930c-e889d9776754/volumes"
Nov 03 20:02:31.372420 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:31.372378    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:35.352071 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:35.352018    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-t9ft8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:02:35.352522 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:35.352105    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-t9ft8" podUID="c8d65667-e957-4334-a43a-71d03f1d1000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:35.420860 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:35.418368    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-bqzwl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:02:35.420860 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:35.418552    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bqzwl" podUID="f5da6be6-a4c0-40f5-8f07-d3de1d188394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:42.023907 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:42.023867    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"]
Nov 03 20:02:42.025748 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:42.025714    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmx5j"]
Nov 03 20:02:43.255298 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:02:43.255259    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:02:43.864397 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:43.864366    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="142001c1-ed1e-430f-8fdf-336807ccab3c" path="/var/lib/kubelet/pods/142001c1-ed1e-430f-8fdf-336807ccab3c/volumes"
Nov 03 20:02:50.668536 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:50.668505    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:51.224235 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:51.224138    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:51.372305 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:51.372253    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:54.830154 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:54.830119    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:54.979167 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:54.979134    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:56.098798 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:56.098759    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:56.271026 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:56.270989    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:56.982211 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:56.982172    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:58.070406 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:58.070368    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:58.810451 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:58.810413    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:59.655015 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:02:59.654925    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:00.347674 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:00.347633    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:01.021089 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:01.021053    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:01.651273 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:01.651230    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:02.356124 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:02.356086    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:03.047602 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:03.047475    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:03.654549 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:03.653138    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:04.391468 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:04.391426    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:05.210487 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:05.210447    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:05.942188 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:05.942145    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:06.578464 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:06.578404    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:07.388328 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:07.388292    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:08.051361 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:08.051308    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:08.762361 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:08.762307    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:09.455489 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:09.455453    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:09.876711 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:09.876671    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:10.248555 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:10.248449    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:10.577168 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:10.577133    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:10.738357 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:10.738320    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:10.934561 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:10.934486    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:11.190973 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:11.190888    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:11.836543 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:11.836510    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:12.529496 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:12.529293    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:13.199863 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.199798    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:13.256541 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:03:13.256502    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:03:13.270892 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.269592    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"]
Nov 03 20:03:13.270892 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.270427    5237 cpu_manager.go:414] "containerMap: removing container" podUID="1ff7e2c7-3ae9-4c64-8769-fec73729b1d0" containerName="cleanup"
Nov 03 20:03:13.270892 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.270446    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff7e2c7-3ae9-4c64-8769-fec73729b1d0" containerName="cleanup"
Nov 03 20:03:13.270892 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.270464    5237 cpu_manager.go:414] "containerMap: removing container" podUID="b143984d-dd0c-4577-ae76-67de0cf11aad" containerName="main"
Nov 03 20:03:13.270892 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.270472    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b143984d-dd0c-4577-ae76-67de0cf11aad" containerName="main"
Nov 03 20:03:13.270892 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.270496    5237 cpu_manager.go:414] "containerMap: removing container" podUID="1ff7e2c7-3ae9-4c64-8769-fec73729b1d0" containerName="cleanup"
Nov 03 20:03:13.270892 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.270504    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff7e2c7-3ae9-4c64-8769-fec73729b1d0" containerName="cleanup"
Nov 03 20:03:13.270892 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.270535    5237 cpu_manager.go:414] "containerMap: removing container" podUID="1ff7e2c7-3ae9-4c64-8769-fec73729b1d0" containerName="cleanup"
Nov 03 20:03:13.270892 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.270542    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff7e2c7-3ae9-4c64-8769-fec73729b1d0" containerName="cleanup"
Nov 03 20:03:13.270892 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.270712    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff7e2c7-3ae9-4c64-8769-fec73729b1d0" containerName="cleanup"
Nov 03 20:03:13.270892 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.270723    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="b143984d-dd0c-4577-ae76-67de0cf11aad" containerName="main"
Nov 03 20:03:13.270892 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.270742    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff7e2c7-3ae9-4c64-8769-fec73729b1d0" containerName="cleanup"
Nov 03 20:03:13.276536 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.274385    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.288226 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.288200    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fcdf83\"/\"kube-root-ca.crt\""
Nov 03 20:03:13.288428 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.288411    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fcdf83\"/\"openshift-service-ca.crt\""
Nov 03 20:03:13.288686 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.288664    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fcdf83\"/\"default-dockercfg-nwg4x\""
Nov 03 20:03:13.288769 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.288669    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fcdf83\"/\"simulator-single-main-fcdf83-kserve-self-signed-certs\""
Nov 03 20:03:13.290257 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.290234    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"]
Nov 03 20:03:13.353436 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.353381    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:13.360354 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.360325    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede13170-c36c-4cb8-87d5-2be0087a4276-tls-certs\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.360487 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.360388    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-model-cache\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.360535 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.360480    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-dshm\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.360606 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.360579    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-tmp-dir\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.360683 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.360667    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l67tm\" (UniqueName: \"kubernetes.io/projected/ede13170-c36c-4cb8-87d5-2be0087a4276-kube-api-access-l67tm\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.360725 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.360692    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-home\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.462246 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.462151    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-model-cache\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.462246 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.462218    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-dshm\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.462619 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.462271    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-tmp-dir\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.462619 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.462337    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l67tm\" (UniqueName: \"kubernetes.io/projected/ede13170-c36c-4cb8-87d5-2be0087a4276-kube-api-access-l67tm\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.462619 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.462363    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-home\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.462619 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.462404    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede13170-c36c-4cb8-87d5-2be0087a4276-tls-certs\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.463774 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.463227    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-tmp-dir\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.464122 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.463911    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-home\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.464122 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.464023    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-model-cache\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.467182 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.467153    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede13170-c36c-4cb8-87d5-2be0087a4276-tls-certs\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.468324 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.468299    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-dshm\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.484768 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.484734    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l67tm\" (UniqueName: \"kubernetes.io/projected/ede13170-c36c-4cb8-87d5-2be0087a4276-kube-api-access-l67tm\") pod \"simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") " pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.590495 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.590461    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:13.877360 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:13.877296    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"]
Nov 03 20:03:13.880803 ip-10-0-133-29 kubenswrapper[5237]: W1103 20:03:13.880769    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podede13170_c36c_4cb8_87d5_2be0087a4276.slice/crio-cf8af6584fff2a07550397e8ed4e022160e31899b2993be6def1676ab659c231 WatchSource:0}: Error finding container cf8af6584fff2a07550397e8ed4e022160e31899b2993be6def1676ab659c231: Status 404 returned error can't find the container with id cf8af6584fff2a07550397e8ed4e022160e31899b2993be6def1676ab659c231
Nov 03 20:03:14.120193 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:14.120154    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:14.514882 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:14.514820    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v" event={"ID":"ede13170-c36c-4cb8-87d5-2be0087a4276","Type":"ContainerStarted","Data":"85913ca71d09c669d281204c7c2bc6c26d1f0ce11fd1eb68590168784d53c4c5"}
Nov 03 20:03:14.515363 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:14.514895    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v" event={"ID":"ede13170-c36c-4cb8-87d5-2be0087a4276","Type":"ContainerStarted","Data":"cf8af6584fff2a07550397e8ed4e022160e31899b2993be6def1676ab659c231"}
Nov 03 20:03:14.536598 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:14.536541    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v" podStartSLOduration=1.5365212499999998 podStartE2EDuration="1.53652125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:14.535604203 +0000 UTC m=+2150.789911430" watchObservedRunningTime="2026-11-03 20:03:14.53652125 +0000 UTC m=+2150.790828475"
Nov 03 20:03:14.591379 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:14.591339    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:14.820045 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:14.819957    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:15.564672 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:15.564634    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:16.339685 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:16.339641    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:17.207614 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:17.207583    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:18.042813 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:18.041251    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:18.790374 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:18.790342    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:19.519655 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:19.519612    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:20.175579 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:20.175532    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:20.878165 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:20.878127    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:21.625862 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:21.619486    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:22.413481 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:22.413442    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:23.039766 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:23.039729    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:23.591080 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:23.591041    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:23.603770 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:23.603737    5237 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:23.739572 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:23.739531    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:24.353508 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:24.353466    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:24.573941 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:24.573909    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:03:25.123314 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:25.123280    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:25.387628 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:25.387545    5237 scope.go:122] "RemoveContainer" containerID="9cb77e6451b655637ce2362c21fe706be1d86748edc81e0b16687168071d691a"
Nov 03 20:03:25.396435 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:25.396412    5237 scope.go:122] "RemoveContainer" containerID="b48ccfc21d0a2d19a6fddc26cad9c85682779e30aade23e2468c40504e24b7d2"
Nov 03 20:03:25.403784 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:25.403762    5237 scope.go:122] "RemoveContainer" containerID="75a0664ea501c902463e00e8d9960062b01dd7058cc1300dd84e49a5204b2865"
Nov 03 20:03:25.831584 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:25.831543    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:26.536090 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:26.536048    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:27.334266 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:27.334230    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:28.060528 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:28.060494    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:28.909006 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:28.908959    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:29.641319 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:29.641284    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:30.471887 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:30.471818    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:31.157504 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:31.156663    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:31.453159 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:31.453063    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:31.932247 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:31.932207    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:32.617729 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:32.617693    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:33.263026 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:33.262978    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:34.039303 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:34.039266    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:34.803106 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:34.803020    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:35.444786 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:35.444739    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:36.168266 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:36.168219    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:37.001419 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:37.001377    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:37.626925 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:37.626882    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:38.337750 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:38.337710    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:39.029334 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:39.029289    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:39.642292 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:39.642254    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:40.023244 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:40.023161    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"]
Nov 03 20:03:40.026449 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:40.026421    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9wj5x"]
Nov 03 20:03:40.461692 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:40.461651    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:41.131128 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:41.131092    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:41.854653 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:41.854615    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:41.862312 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:41.862278    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5843167c-f9fb-4179-b2d4-e43bae846bbc" path="/var/lib/kubelet/pods/5843167c-f9fb-4179-b2d4-e43bae846bbc/volumes"
Nov 03 20:03:42.572003 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:42.571965    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:43.244534 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:43.241298    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:43.270756 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:03:43.270718    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:03:43.934262 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:43.934202    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:44.698806 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:44.698772    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:45.349208 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:45.349172    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:45.953796 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:45.953759    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:46.636892 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:46.636853    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:47.330938 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:47.330888    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:48.144002 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:48.137815    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:48.867739 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:48.867702    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:49.756810 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:49.756777    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:50.443874 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:50.443822    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:51.157662 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:51.156628    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:51.843306 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:51.842147    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:52.452195 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:52.452158    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:53.129571 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:53.129536    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:53.318398 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:53.318359    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:53.953039 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:53.953008    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:54.659097 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:54.659047    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:55.375132 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:55.375093    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:56.037532 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:56.037500    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:56.667235 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:56.667201    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:57.460615 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:57.455463    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:58.134453 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:58.134406    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"]
Nov 03 20:03:58.237933 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:58.237899    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:58.418148 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:58.418059    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:59.058010 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:59.057966    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:59.733709 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:03:59.733670    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:00.382789 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:00.382751    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:01.021761 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:01.021723    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:01.629794 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:01.629755    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:02.315292 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:02.315251    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:02.956232 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:02.956196    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:03.569137 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:03.569106    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:03.738432 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:03.738397    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:04.197215 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:04.197179    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:04.335307 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:04.335270    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:04.465277 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:04.465198    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:04.941686 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:04.941651    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:11.389059 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:11.389019    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:11.515735 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:11.515689    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:11.768788 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:11.768697    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:13.154142 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.154100    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v" podUID="ede13170-c36c-4cb8-87d5-2be0087a4276" containerName="main" containerID="cri-o://85913ca71d09c669d281204c7c2bc6c26d1f0ce11fd1eb68590168784d53c4c5" gracePeriod=45
Nov 03 20:04:13.256030 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:04:13.255990    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:04:13.394569 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.394542    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:04:13.540171 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.540142    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-home\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-home\") pod \"ede13170-c36c-4cb8-87d5-2be0087a4276\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") "
Nov 03 20:04:13.540372 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.540200    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-model-cache\") pod \"ede13170-c36c-4cb8-87d5-2be0087a4276\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") "
Nov 03 20:04:13.540372 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.540253    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede13170-c36c-4cb8-87d5-2be0087a4276-kube-api-access-l67tm\" (UniqueName: \"kubernetes.io/projected/ede13170-c36c-4cb8-87d5-2be0087a4276-kube-api-access-l67tm\") pod \"ede13170-c36c-4cb8-87d5-2be0087a4276\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") "
Nov 03 20:04:13.540372 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.540297    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-dshm\") pod \"ede13170-c36c-4cb8-87d5-2be0087a4276\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") "
Nov 03 20:04:13.540372 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.540325    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-tmp-dir\") pod \"ede13170-c36c-4cb8-87d5-2be0087a4276\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") "
Nov 03 20:04:13.540599 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.540375    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede13170-c36c-4cb8-87d5-2be0087a4276-tls-certs\" (UniqueName: \"kubernetes.io/secret/ede13170-c36c-4cb8-87d5-2be0087a4276-tls-certs\") pod \"ede13170-c36c-4cb8-87d5-2be0087a4276\" (UID: \"ede13170-c36c-4cb8-87d5-2be0087a4276\") "
Nov 03 20:04:13.540599 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.540482    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-model-cache" pod "ede13170-c36c-4cb8-87d5-2be0087a4276" (UID: "ede13170-c36c-4cb8-87d5-2be0087a4276"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:13.540599 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.540488    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-home" pod "ede13170-c36c-4cb8-87d5-2be0087a4276" (UID: "ede13170-c36c-4cb8-87d5-2be0087a4276"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:13.540755 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.540654    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-tmp-dir" pod "ede13170-c36c-4cb8-87d5-2be0087a4276" (UID: "ede13170-c36c-4cb8-87d5-2be0087a4276"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:13.540755 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.540705    5237 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-home\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:13.540755 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.540726    5237 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-model-cache\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:13.542586 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.542563    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede13170-c36c-4cb8-87d5-2be0087a4276-tls-certs" pod "ede13170-c36c-4cb8-87d5-2be0087a4276" (UID: "ede13170-c36c-4cb8-87d5-2be0087a4276"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:13.542710 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.542603    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-dshm" pod "ede13170-c36c-4cb8-87d5-2be0087a4276" (UID: "ede13170-c36c-4cb8-87d5-2be0087a4276"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:13.542710 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.542609    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede13170-c36c-4cb8-87d5-2be0087a4276-kube-api-access-l67tm" pod "ede13170-c36c-4cb8-87d5-2be0087a4276" (UID: "ede13170-c36c-4cb8-87d5-2be0087a4276"). InnerVolumeSpecName "kube-api-access-l67tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:13.642122 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.642081    5237 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede13170-c36c-4cb8-87d5-2be0087a4276-tls-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:13.642122 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.642117    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l67tm\" (UniqueName: \"kubernetes.io/projected/ede13170-c36c-4cb8-87d5-2be0087a4276-kube-api-access-l67tm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:13.642122 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.642126    5237 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-dshm\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:13.642367 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.642139    5237 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede13170-c36c-4cb8-87d5-2be0087a4276-tmp-dir\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:13.797563 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.797479    5237 generic.go:356] "Generic (PLEG): container finished" podID="ede13170-c36c-4cb8-87d5-2be0087a4276" containerID="85913ca71d09c669d281204c7c2bc6c26d1f0ce11fd1eb68590168784d53c4c5" exitCode=0
Nov 03 20:04:13.797563 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.797524    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"
Nov 03 20:04:13.797750 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.797562    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v" event={"ID":"ede13170-c36c-4cb8-87d5-2be0087a4276","Type":"ContainerDied","Data":"85913ca71d09c669d281204c7c2bc6c26d1f0ce11fd1eb68590168784d53c4c5"}
Nov 03 20:04:13.797750 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.797604    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v" event={"ID":"ede13170-c36c-4cb8-87d5-2be0087a4276","Type":"ContainerDied","Data":"cf8af6584fff2a07550397e8ed4e022160e31899b2993be6def1676ab659c231"}
Nov 03 20:04:13.797750 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.797622    5237 scope.go:122] "RemoveContainer" containerID="85913ca71d09c669d281204c7c2bc6c26d1f0ce11fd1eb68590168784d53c4c5"
Nov 03 20:04:13.808769 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.808743    5237 scope.go:122] "RemoveContainer" containerID="85913ca71d09c669d281204c7c2bc6c26d1f0ce11fd1eb68590168784d53c4c5"
Nov 03 20:04:13.809067 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:04:13.809046    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85913ca71d09c669d281204c7c2bc6c26d1f0ce11fd1eb68590168784d53c4c5\": container with ID starting with 85913ca71d09c669d281204c7c2bc6c26d1f0ce11fd1eb68590168784d53c4c5 not found: ID does not exist" containerID="85913ca71d09c669d281204c7c2bc6c26d1f0ce11fd1eb68590168784d53c4c5"
Nov 03 20:04:13.809160 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.809074    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85913ca71d09c669d281204c7c2bc6c26d1f0ce11fd1eb68590168784d53c4c5"} err="failed to get container status \"85913ca71d09c669d281204c7c2bc6c26d1f0ce11fd1eb68590168784d53c4c5\": rpc error: code = NotFound desc = could not find container \"85913ca71d09c669d281204c7c2bc6c26d1f0ce11fd1eb68590168784d53c4c5\": container with ID starting with 85913ca71d09c669d281204c7c2bc6c26d1f0ce11fd1eb68590168784d53c4c5 not found: ID does not exist"
Nov 03 20:04:13.827399 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.827367    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"]
Nov 03 20:04:13.831764 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.831739    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-fcdf83/simulator-single-main-fcdf83-kserve-5b4ff56c5d-pxq2v"]
Nov 03 20:04:13.858431 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:13.858400    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede13170-c36c-4cb8-87d5-2be0087a4276" path="/var/lib/kubelet/pods/ede13170-c36c-4cb8-87d5-2be0087a4276/volumes"
Nov 03 20:04:17.188584 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:17.188541    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:25.428681 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:25.428647    5237 scope.go:122] "RemoveContainer" containerID="486a03aada27a948a4c25c5313abacf4876e319d83ed015ddd705e530de5091f"
Nov 03 20:04:25.436728 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:25.436705    5237 scope.go:122] "RemoveContainer" containerID="ac2acfa47b1b7f1712c66b22a6d15836e6770c2057ea3ac857dbac0f979455ca"
Nov 03 20:04:25.444332 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:25.444314    5237 scope.go:122] "RemoveContainer" containerID="0bfd04c7c08f8052d4926f65dc493aa1f64d2ec6bba2c28aca4dead6d8636adf"
Nov 03 20:04:31.514166 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:31.514105    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:32.385814 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:32.385775    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:35.595300 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:35.595243    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:36.345198 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:36.345163    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:41.874271 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:04:41.874232    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:43.254902 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:04:43.254866    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:05:00.526999 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:00.526953    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:01.857900 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:01.854778    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:03.189109 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:03.189053    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:13.254249 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:05:13.254209    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:05:13.994032 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:13.993980    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:25.471134 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:25.471096    5237 scope.go:122] "RemoveContainer" containerID="4898c1d88826d514951b3870f7d47d67e74ab880a5ac23f2a47e19df5a6a3676"
Nov 03 20:05:31.559066 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:31.559021    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:35.943289 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:35.943244    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"]
Nov 03 20:05:35.943774 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:35.943727    5237 cpu_manager.go:414] "containerMap: removing container" podUID="ede13170-c36c-4cb8-87d5-2be0087a4276" containerName="main"
Nov 03 20:05:35.943774 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:35.943745    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede13170-c36c-4cb8-87d5-2be0087a4276" containerName="main"
Nov 03 20:05:35.943932 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:35.943910    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede13170-c36c-4cb8-87d5-2be0087a4276" containerName="main"
Nov 03 20:05:35.943932 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:35.943927    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff7e2c7-3ae9-4c64-8769-fec73729b1d0" containerName="cleanup"
Nov 03 20:05:35.948280 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:35.948259    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:35.955788 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:35.955192    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 20:05:35.958812 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:35.957423    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"]
Nov 03 20:05:35.958812 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:35.957929    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wm9nt\""
Nov 03 20:05:35.958812 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:35.958151    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:05:35.958812 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:35.958225    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:05:35.968746 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:35.968714    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac9dbdbd-6b70-4422-b715-2cc79394ff20-metrics-certs\") pod \"maas-api-57c44bc9cc-9mrpd\" (UID: \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\") " pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:35.968896 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:35.968759    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac9dbdbd-6b70-4422-b715-2cc79394ff20-maas-api-tls\") pod \"maas-api-57c44bc9cc-9mrpd\" (UID: \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\") " pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:35.969000 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:35.968981    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac9dbdbd-6b70-4422-b715-2cc79394ff20-openshift-service-ca\") pod \"maas-api-57c44bc9cc-9mrpd\" (UID: \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\") " pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:35.969059 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:35.969023    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9ldz\" (UniqueName: \"kubernetes.io/projected/ac9dbdbd-6b70-4422-b715-2cc79394ff20-kube-api-access-d9ldz\") pod \"maas-api-57c44bc9cc-9mrpd\" (UID: \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\") " pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:36.055447 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:36.055419    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 20:05:36.069821 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:36.069793    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac9dbdbd-6b70-4422-b715-2cc79394ff20-openshift-service-ca\") pod \"maas-api-57c44bc9cc-9mrpd\" (UID: \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\") " pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:36.069973 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:36.069827    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9ldz\" (UniqueName: \"kubernetes.io/projected/ac9dbdbd-6b70-4422-b715-2cc79394ff20-kube-api-access-d9ldz\") pod \"maas-api-57c44bc9cc-9mrpd\" (UID: \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\") " pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:36.069973 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:36.069885    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac9dbdbd-6b70-4422-b715-2cc79394ff20-metrics-certs\") pod \"maas-api-57c44bc9cc-9mrpd\" (UID: \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\") " pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:36.069973 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:36.069903    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac9dbdbd-6b70-4422-b715-2cc79394ff20-maas-api-tls\") pod \"maas-api-57c44bc9cc-9mrpd\" (UID: \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\") " pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:36.070520 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:36.070501    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac9dbdbd-6b70-4422-b715-2cc79394ff20-openshift-service-ca\") pod \"maas-api-57c44bc9cc-9mrpd\" (UID: \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\") " pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:36.072387 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:36.072359    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac9dbdbd-6b70-4422-b715-2cc79394ff20-metrics-certs\") pod \"maas-api-57c44bc9cc-9mrpd\" (UID: \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\") " pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:36.072528 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:36.072509    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac9dbdbd-6b70-4422-b715-2cc79394ff20-maas-api-tls\") pod \"maas-api-57c44bc9cc-9mrpd\" (UID: \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\") " pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:36.082303 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:36.082278    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9ldz\" (UniqueName: \"kubernetes.io/projected/ac9dbdbd-6b70-4422-b715-2cc79394ff20-kube-api-access-d9ldz\") pod \"maas-api-57c44bc9cc-9mrpd\" (UID: \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\") " pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:36.263009 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:36.262932    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:36.538944 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:36.538904    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"]
Nov 03 20:05:36.542688 ip-10-0-133-29 kubenswrapper[5237]: W1103 20:05:36.542660    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9dbdbd_6b70_4422_b715_2cc79394ff20.slice/crio-27968d25d35cc232cbf5dd9edb5a12564180ecb03d4252597e3e0c68083309da WatchSource:0}: Error finding container 27968d25d35cc232cbf5dd9edb5a12564180ecb03d4252597e3e0c68083309da: Status 404 returned error can't find the container with id 27968d25d35cc232cbf5dd9edb5a12564180ecb03d4252597e3e0c68083309da
Nov 03 20:05:36.544167 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:36.544148    5237 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:05:37.275980 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:37.275943    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd" event={"ID":"ac9dbdbd-6b70-4422-b715-2cc79394ff20","Type":"ContainerStarted","Data":"19a24dc89f6e6eca39a4babf7a73d1695bcc415258328acd067ad07a16ffade7"}
Nov 03 20:05:37.276375 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:37.275993    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd" event={"ID":"ac9dbdbd-6b70-4422-b715-2cc79394ff20","Type":"ContainerStarted","Data":"27968d25d35cc232cbf5dd9edb5a12564180ecb03d4252597e3e0c68083309da"}
Nov 03 20:05:37.276375 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:37.276082    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:37.296871 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:37.296746    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd" podStartSLOduration=1.800095014 podStartE2EDuration="2.296730633s" totalImagesPullingTime="496.635619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:37.295538575 +0000 UTC m=+2293.549845801" watchObservedRunningTime="2026-11-03 20:05:37.296730633 +0000 UTC m=+2293.551037861"
Nov 03 20:05:43.014265 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.014229    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:43.255253 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:05:43.255208    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:05:43.286584 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.286561    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:43.563218 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.563137    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"]
Nov 03 20:05:43.567409 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.567370    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"
Nov 03 20:05:43.577931 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.577897    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"]
Nov 03 20:05:43.588289 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.588252    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:43.642622 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.642591    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/038db3ee-a251-4d26-bd5a-e4a060f1b157-maas-api-tls\") pod \"maas-api-755bc5f869-khk9v\" (UID: \"038db3ee-a251-4d26-bd5a-e4a060f1b157\") " pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"
Nov 03 20:05:43.642850 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.642723    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl2jh\" (UniqueName: \"kubernetes.io/projected/038db3ee-a251-4d26-bd5a-e4a060f1b157-kube-api-access-cl2jh\") pod \"maas-api-755bc5f869-khk9v\" (UID: \"038db3ee-a251-4d26-bd5a-e4a060f1b157\") " pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"
Nov 03 20:05:43.642850 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.642760    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/038db3ee-a251-4d26-bd5a-e4a060f1b157-openshift-service-ca\") pod \"maas-api-755bc5f869-khk9v\" (UID: \"038db3ee-a251-4d26-bd5a-e4a060f1b157\") " pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"
Nov 03 20:05:43.642850 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.642790    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/038db3ee-a251-4d26-bd5a-e4a060f1b157-metrics-certs\") pod \"maas-api-755bc5f869-khk9v\" (UID: \"038db3ee-a251-4d26-bd5a-e4a060f1b157\") " pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"
Nov 03 20:05:43.727128 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.727090    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:43.744238 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.744200    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl2jh\" (UniqueName: \"kubernetes.io/projected/038db3ee-a251-4d26-bd5a-e4a060f1b157-kube-api-access-cl2jh\") pod \"maas-api-755bc5f869-khk9v\" (UID: \"038db3ee-a251-4d26-bd5a-e4a060f1b157\") " pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"
Nov 03 20:05:43.744238 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.744243    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/038db3ee-a251-4d26-bd5a-e4a060f1b157-openshift-service-ca\") pod \"maas-api-755bc5f869-khk9v\" (UID: \"038db3ee-a251-4d26-bd5a-e4a060f1b157\") " pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"
Nov 03 20:05:43.744484 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.744276    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/038db3ee-a251-4d26-bd5a-e4a060f1b157-metrics-certs\") pod \"maas-api-755bc5f869-khk9v\" (UID: \"038db3ee-a251-4d26-bd5a-e4a060f1b157\") " pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"
Nov 03 20:05:43.744484 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.744304    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/038db3ee-a251-4d26-bd5a-e4a060f1b157-maas-api-tls\") pod \"maas-api-755bc5f869-khk9v\" (UID: \"038db3ee-a251-4d26-bd5a-e4a060f1b157\") " pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"
Nov 03 20:05:43.744917 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.744892    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/038db3ee-a251-4d26-bd5a-e4a060f1b157-openshift-service-ca\") pod \"maas-api-755bc5f869-khk9v\" (UID: \"038db3ee-a251-4d26-bd5a-e4a060f1b157\") " pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"
Nov 03 20:05:43.746773 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.746750    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/038db3ee-a251-4d26-bd5a-e4a060f1b157-metrics-certs\") pod \"maas-api-755bc5f869-khk9v\" (UID: \"038db3ee-a251-4d26-bd5a-e4a060f1b157\") " pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"
Nov 03 20:05:43.746910 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.746788    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/038db3ee-a251-4d26-bd5a-e4a060f1b157-maas-api-tls\") pod \"maas-api-755bc5f869-khk9v\" (UID: \"038db3ee-a251-4d26-bd5a-e4a060f1b157\") " pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"
Nov 03 20:05:43.756122 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.756098    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl2jh\" (UniqueName: \"kubernetes.io/projected/038db3ee-a251-4d26-bd5a-e4a060f1b157-kube-api-access-cl2jh\") pod \"maas-api-755bc5f869-khk9v\" (UID: \"038db3ee-a251-4d26-bd5a-e4a060f1b157\") " pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"
Nov 03 20:05:43.883553 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:43.883471    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"
Nov 03 20:05:44.158399 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:44.158295    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"]
Nov 03 20:05:44.161252 ip-10-0-133-29 kubenswrapper[5237]: W1103 20:05:44.161231    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038db3ee_a251_4d26_bd5a_e4a060f1b157.slice/crio-bcdec863315c8440a1fe0413f822d50391a8e405298e30a073410cebf5a43878 WatchSource:0}: Error finding container bcdec863315c8440a1fe0413f822d50391a8e405298e30a073410cebf5a43878: Status 404 returned error can't find the container with id bcdec863315c8440a1fe0413f822d50391a8e405298e30a073410cebf5a43878
Nov 03 20:05:44.306416 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:44.306383    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v" event={"ID":"038db3ee-a251-4d26-bd5a-e4a060f1b157","Type":"ContainerStarted","Data":"bcdec863315c8440a1fe0413f822d50391a8e405298e30a073410cebf5a43878"}
Nov 03 20:05:45.312169 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:45.312131    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v" event={"ID":"038db3ee-a251-4d26-bd5a-e4a060f1b157","Type":"ContainerStarted","Data":"980f710429cab80c597f0076482a636237ce5e60af791923950a41776e539900"}
Nov 03 20:05:45.312592 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:45.312304    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"
Nov 03 20:05:45.332876 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:45.332802    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v" podStartSLOduration=1.93101122 podStartE2EDuration="2.332788417s" totalImagesPullingTime="401.777197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:45.332368094 +0000 UTC m=+2301.586675318" watchObservedRunningTime="2026-11-03 20:05:45.332788417 +0000 UTC m=+2301.587095643"
Nov 03 20:05:46.077306 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:46.077266    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:46.219440 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:46.219403    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:47.325881 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:47.325848    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:47.494986 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:47.494891    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:48.145511 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:48.145470    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:48.952071 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:48.952034    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:49.621807 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:49.621771    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:50.330237 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:50.330201    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:50.985444 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:50.985405    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:51.322424 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:51.322393    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-755bc5f869-khk9v"
Nov 03 20:05:51.379546 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:51.379509    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"]
Nov 03 20:05:51.380183 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:51.379740    5237 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd" podUID="ac9dbdbd-6b70-4422-b715-2cc79394ff20" containerName="maas-api" containerID="cri-o://19a24dc89f6e6eca39a4babf7a73d1695bcc415258328acd067ad07a16ffade7" gracePeriod=30
Nov 03 20:05:51.629152 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:51.629110    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:51.654565 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:51.654535    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:51.721363 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:51.721328    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac9dbdbd-6b70-4422-b715-2cc79394ff20-kube-api-access-d9ldz\" (UniqueName: \"kubernetes.io/projected/ac9dbdbd-6b70-4422-b715-2cc79394ff20-kube-api-access-d9ldz\") pod \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\" (UID: \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\") "
Nov 03 20:05:51.721578 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:51.721379    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac9dbdbd-6b70-4422-b715-2cc79394ff20-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac9dbdbd-6b70-4422-b715-2cc79394ff20-maas-api-tls\") pod \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\" (UID: \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\") "
Nov 03 20:05:51.721578 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:51.721471    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac9dbdbd-6b70-4422-b715-2cc79394ff20-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac9dbdbd-6b70-4422-b715-2cc79394ff20-openshift-service-ca\") pod \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\" (UID: \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\") "
Nov 03 20:05:51.721578 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:51.721539    5237 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac9dbdbd-6b70-4422-b715-2cc79394ff20-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac9dbdbd-6b70-4422-b715-2cc79394ff20-metrics-certs\") pod \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\" (UID: \"ac9dbdbd-6b70-4422-b715-2cc79394ff20\") "
Nov 03 20:05:51.721884 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:51.721821    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac9dbdbd-6b70-4422-b715-2cc79394ff20-openshift-service-ca" pod "ac9dbdbd-6b70-4422-b715-2cc79394ff20" (UID: "ac9dbdbd-6b70-4422-b715-2cc79394ff20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:51.723577 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:51.723553    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac9dbdbd-6b70-4422-b715-2cc79394ff20-maas-api-tls" pod "ac9dbdbd-6b70-4422-b715-2cc79394ff20" (UID: "ac9dbdbd-6b70-4422-b715-2cc79394ff20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:51.723695 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:51.723573    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9dbdbd-6b70-4422-b715-2cc79394ff20-kube-api-access-d9ldz" pod "ac9dbdbd-6b70-4422-b715-2cc79394ff20" (UID: "ac9dbdbd-6b70-4422-b715-2cc79394ff20"). InnerVolumeSpecName "kube-api-access-d9ldz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:51.723695 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:51.723588    5237 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac9dbdbd-6b70-4422-b715-2cc79394ff20-metrics-certs" pod "ac9dbdbd-6b70-4422-b715-2cc79394ff20" (UID: "ac9dbdbd-6b70-4422-b715-2cc79394ff20"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:51.822753 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:51.822712    5237 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac9dbdbd-6b70-4422-b715-2cc79394ff20-metrics-certs\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:51.822753 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:51.822745    5237 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9ldz\" (UniqueName: \"kubernetes.io/projected/ac9dbdbd-6b70-4422-b715-2cc79394ff20-kube-api-access-d9ldz\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:51.822753 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:51.822759    5237 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac9dbdbd-6b70-4422-b715-2cc79394ff20-maas-api-tls\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:51.823051 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:51.822772    5237 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac9dbdbd-6b70-4422-b715-2cc79394ff20-openshift-service-ca\") on node \"ip-10-0-133-29.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:52.344319 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:52.344281    5237 generic.go:356] "Generic (PLEG): container finished" podID="ac9dbdbd-6b70-4422-b715-2cc79394ff20" containerID="19a24dc89f6e6eca39a4babf7a73d1695bcc415258328acd067ad07a16ffade7" exitCode=0
Nov 03 20:05:52.344504 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:52.344334    5237 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"
Nov 03 20:05:52.344504 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:52.344371    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd" event={"ID":"ac9dbdbd-6b70-4422-b715-2cc79394ff20","Type":"ContainerDied","Data":"19a24dc89f6e6eca39a4babf7a73d1695bcc415258328acd067ad07a16ffade7"}
Nov 03 20:05:52.344504 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:52.344429    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd" event={"ID":"ac9dbdbd-6b70-4422-b715-2cc79394ff20","Type":"ContainerDied","Data":"27968d25d35cc232cbf5dd9edb5a12564180ecb03d4252597e3e0c68083309da"}
Nov 03 20:05:52.344504 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:52.344452    5237 scope.go:122] "RemoveContainer" containerID="19a24dc89f6e6eca39a4babf7a73d1695bcc415258328acd067ad07a16ffade7"
Nov 03 20:05:52.353095 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:52.353077    5237 scope.go:122] "RemoveContainer" containerID="19a24dc89f6e6eca39a4babf7a73d1695bcc415258328acd067ad07a16ffade7"
Nov 03 20:05:52.353365 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:05:52.353345    5237 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19a24dc89f6e6eca39a4babf7a73d1695bcc415258328acd067ad07a16ffade7\": container with ID starting with 19a24dc89f6e6eca39a4babf7a73d1695bcc415258328acd067ad07a16ffade7 not found: ID does not exist" containerID="19a24dc89f6e6eca39a4babf7a73d1695bcc415258328acd067ad07a16ffade7"
Nov 03 20:05:52.353420 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:52.353374    5237 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a24dc89f6e6eca39a4babf7a73d1695bcc415258328acd067ad07a16ffade7"} err="failed to get container status \"19a24dc89f6e6eca39a4babf7a73d1695bcc415258328acd067ad07a16ffade7\": rpc error: code = NotFound desc = could not find container \"19a24dc89f6e6eca39a4babf7a73d1695bcc415258328acd067ad07a16ffade7\": container with ID starting with 19a24dc89f6e6eca39a4babf7a73d1695bcc415258328acd067ad07a16ffade7 not found: ID does not exist"
Nov 03 20:05:52.361518 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:52.361493    5237 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"]
Nov 03 20:05:52.365669 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:52.365642    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:52.366225 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:52.366206    5237 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-57c44bc9cc-9mrpd"]
Nov 03 20:05:52.940753 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:52.940722    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:53.540704 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:53.540671    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:53.669287 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:53.669245    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:53.858904 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:53.858802    5237 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac9dbdbd-6b70-4422-b715-2cc79394ff20" path="/var/lib/kubelet/pods/ac9dbdbd-6b70-4422-b715-2cc79394ff20/volumes"
Nov 03 20:05:54.145334 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:54.145231    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:54.885958 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:54.885912    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:55.626779 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:55.626732    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:56.249586 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:56.249547    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:56.919911 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:56.919870    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:57.633398 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:57.633364    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:58.310061 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:58.310019    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:59.022295 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:59.022252    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:59.623847 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:05:59.623797    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:00.280497 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:00.280447    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:00.927739 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:00.927680    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:01.649013 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:01.648967    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:02.233847 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:02.233806    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:02.947141 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:02.947107    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:03.606232 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:03.606190    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:04.320372 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:04.320339    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:04.928559 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:04.928528    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:05.552374 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:05.552303    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:06.221161 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:06.221122    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:06.384006 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:06.383975    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:06.903284 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:06.903247    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:07.119762 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:07.119720    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:07.249222 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:07.249129    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:07.528490 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:07.528404    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:07.694116 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:07.694084    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:07.825490 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:07.825450    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:13.256708 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:06:13.256661    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:06:13.280592 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:13.280534    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:14.576122 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:14.576071    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:15.021216 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:15.021125    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:15.270757 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:15.270721    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:17.892641 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:17.892588    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:18.041680 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:18.041641    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:19.099412 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:19.099374    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:19.241849 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:19.241799    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:19.400484 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:19.400397    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:19.874526 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:19.874410    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:20.671105 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:20.671065    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:21.430104 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:21.430054    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:22.231802 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:22.231764    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:22.947480 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:22.947441    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:23.538636 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:23.538598    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:24.233980 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:24.233928    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:25.020107 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:25.020067    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:25.630937 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:25.630901    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:26.236068 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:26.236035    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:26.835030 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:26.834994    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:27.429520 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:27.429484    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:28.119277 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:28.119237    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:28.866761 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:28.866677    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:29.448352 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:29.448316    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:30.134514 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:30.134421    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:30.732774 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:30.732736    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:31.375276 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:31.375239    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:31.608001 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:31.607963    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:32.054689 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:32.054573    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:32.771083 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:32.770917    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:32.960817 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:32.960778    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:33.463144 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:33.463084    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:34.238454 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:34.238416    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:34.918729 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:34.918636    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:35.544302 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:35.544243    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:36.237089 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:36.237047    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:36.857328 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:36.857283    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:37.515485 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:37.515445    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:38.136275 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:38.136227    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:38.888720 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:38.888676    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:39.280594 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:39.280507    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:39.522437 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:39.522395    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:40.237034 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:40.237003    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:41.058384 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:41.058350    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:41.648323 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:41.648257    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:42.229723 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:42.229687    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:43.124160 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:43.124121    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:43.255754 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:06:43.255717    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:06:43.734214 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:43.734172    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:44.419285 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:44.419251    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:45.187996 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:45.187958    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:45.821778 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:45.821735    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:46.518042 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:46.517982    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:47.122609 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:47.122563    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:47.751813 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:47.749377    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:48.506048 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:48.506007    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:49.124502 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:49.124464    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:49.728863 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:49.728804    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:50.402228 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:50.402192    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:51.045717 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:51.045671    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:51.709262 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:51.709227    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:52.322487 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:52.322451    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:52.948590 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:52.948551    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:53.547934 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:53.547882    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:54.146442 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:54.146395    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:54.727765 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:54.727728    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:55.332600 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:55.332566    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:56.020991 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:56.020955    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:56.630520 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:56.630484    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:57.235055 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:57.235020    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:57.838022 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:57.837986    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:58.480644 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:58.480605    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:59.137975 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:59.137930    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:59.718170 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:06:59.718134    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:00.314406 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:00.314356    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:00.942009 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:00.941971    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:01.546964 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:01.546925    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:02.218309 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:02.218264    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:02.869153 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:02.869122    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:03.449762 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:03.449724    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:04.035324 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:04.035284    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:04.629063 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:04.629027    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:05.224875 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:05.224818    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:05.928316 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:05.928281    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:06.521308 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:06.521270    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:07.135579 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:07.135538    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:08.576262 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:08.576217    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:09.024477 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:09.024374    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:09.631471 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:09.631430    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:10.317015 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:10.316981    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:11.023542 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:11.023487    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:11.621002 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:11.620969    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:12.339493 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:12.339452    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:13.010760 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:13.010727    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:13.256070 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:07:13.256027    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:07:13.622539 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:13.622500    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:14.334545 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:14.334508    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:15.028197 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:15.028159    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:15.629164 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:15.629116    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:16.247605 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:16.247546    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:16.830008 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:16.829976    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:17.444606 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:17.444565    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:18.026635 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:18.026596    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:18.669709 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:18.669675    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:19.247257 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:19.247207    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:19.831722 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:19.831688    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:20.625498 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:20.625459    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:21.250932 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:21.250894    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:21.922814 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:21.922779    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:22.549522 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:22.549482    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:23.219553 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:23.219509    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:23.846623 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:23.846587    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:24.442389 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:24.442345    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:25.042643 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:25.042605    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:25.737329 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:25.737291    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:26.355780 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:26.355743    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:27.028358 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:27.028319    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:27.645731 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:27.645692    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:28.248377 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:28.248335    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:28.925918 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:28.925883    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:29.539365 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:29.539323    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:30.132627 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:30.132594    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:30.816144 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:30.816108    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:31.455527 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:31.455481    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:31.735568 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:31.735486    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:32.136975 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:32.136922    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:32.864699 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:32.864656    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:33.449495 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:33.449458    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:34.043652 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:34.043615    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:34.641454 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:34.641411    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:35.238726 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:35.238682    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:35.915735 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:35.915701    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:36.516223 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:36.516182    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:37.115518 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:37.115479    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:37.749545 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:37.747385    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:38.413119 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:38.413067    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:39.029740 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:39.029704    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:39.632278 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:39.632232    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:40.344670 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:40.344631    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:40.927688 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:40.927648    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:41.549789 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:41.549754    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:42.216983 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:42.216934    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:42.847750 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:42.847713    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:43.255002 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:07:43.254914    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:07:43.523929 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:43.523814    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:44.231335 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:44.231302    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:44.972644 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:44.972596    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:45.645120 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:45.645060    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:46.323148 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:46.322880    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:46.953742 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:46.953703    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:47.630419 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:47.630384    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:48.238108 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:48.238071    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:48.848109 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:48.848071    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:49.547700 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:49.547067    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:50.138368 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:50.138333    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:50.787219 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:50.787186    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:51.422712 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:51.422675    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:52.023441 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:52.023394    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:52.641204 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:52.641168    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:53.411484 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:53.411443    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:54.132888 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:54.132824    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:54.744965 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:54.744929    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:55.340442 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:55.340409    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:55.944393 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:55.944360    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:56.527908 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:56.527868    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:57.291366 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:57.291321    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:57.943509 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:57.943470    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:58.531595 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:58.531542    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:59.148070 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:59.148019    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:59.716687 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:07:59.716646    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:00.325259 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:00.325220    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:00.921474 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:00.921423    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:01.546338 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:01.546309    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:02.214718 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:02.214682    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:02.823165 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:02.823127    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:03.307937 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:03.307900    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:03.439094 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:03.439053    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:03.575535 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:03.575379    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:09.062460 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:09.062412    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:09.193411 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:09.193360    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:09.324643 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:09.324561    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:09.806304 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:09.806267    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:13.254697 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:08:13.254664    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:08:15.866483 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:15.866448    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:15.993939 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:15.993896    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:16.264528 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:16.264437    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:17.930371 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:17.930337    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:18.064221 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:18.064185    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:31.720162 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:31.720127    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:43.255021 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:08:43.254982    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:08:58.310047 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:58.310007    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:58.450191 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:08:58.450148    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:11.237026 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:11.236970    5237 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-f5xfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:09:11.237621 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:11.237063    5237 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-f5xfb" podUID="98ec8c1c-49db-492e-958e-fabe09d044fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:09:13.255346 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:09:13.255308    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:09:21.833404 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:21.833363    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-msxfh_8e2a098b-1b62-4639-bd21-bdf36b47db70/manager/0.log"
Nov 03 20:09:22.133119 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:22.133028    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-694f8bb88d-h7fff_9a72f77e-2af9-4e75-bbfe-a20447358ef5/manager/0.log"
Nov 03 20:09:22.335248 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:22.335217    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-78kwl_f4b0fa79-e38f-4966-9c9f-e070a4091196/server/0.log"
Nov 03 20:09:22.752451 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:22.752418    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4gb64_f816ed7b-55b4-4f23-9fd7-28851ef00f1c/proxy/0.log"
Nov 03 20:09:23.544801 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:23.544770    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-755bc5f869-khk9v_038db3ee-a251-4d26-bd5a-e4a060f1b157/maas-api/0.log"
Nov 03 20:09:26.890613 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:26.890575    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-m8n7h_e2402039-dd0c-47b4-be42-1edef761c1e6/manager/0.log"
Nov 03 20:09:27.283299 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:27.283218    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-csmwj_c4376e17-400a-4487-bdff-ef9878f40a0e/kuadrant-console-plugin/0.log"
Nov 03 20:09:28.968347 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:28.968310    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2prbp_f7abe1b7-d7bc-402e-a762-e35a95520eb3/discovery/0.log"
Nov 03 20:09:29.356673 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:29.356641    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b4fd54d5-wn7tv_3e5c9e63-14b4-49b8-b8be-44639d2e8d9e/kube-auth-proxy/0.log"
Nov 03 20:09:29.737293 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:29.737213    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-bqzwl_f5da6be6-a4c0-40f5-8f07-d3de1d188394/istio-proxy/0.log"
Nov 03 20:09:30.119579 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:30.119544    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-f5xfb_98ec8c1c-49db-492e-958e-fabe09d044fb/istio-proxy/0.log"
Nov 03 20:09:31.151346 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:31.151309    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-t9ft8_c8d65667-e957-4334-a43a-71d03f1d1000/istio-proxy/0.log"
Nov 03 20:09:31.767759 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:31.767729    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:43.255501 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:09:43.255416    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:09:45.657934 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:45.657902    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-msxfh_8e2a098b-1b62-4639-bd21-bdf36b47db70/manager/0.log"
Nov 03 20:09:45.878553 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:45.878518    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-694f8bb88d-h7fff_9a72f77e-2af9-4e75-bbfe-a20447358ef5/manager/0.log"
Nov 03 20:09:46.099121 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:46.099084    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-78kwl_f4b0fa79-e38f-4966-9c9f-e070a4091196/server/0.log"
Nov 03 20:09:46.567674 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:46.567631    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4gb64_f816ed7b-55b4-4f23-9fd7-28851ef00f1c/proxy/0.log"
Nov 03 20:09:47.126730 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:47.126703    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-755bc5f869-khk9v_038db3ee-a251-4d26-bd5a-e4a060f1b157/maas-api/0.log"
Nov 03 20:09:50.205190 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:50.205155    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-m8n7h_e2402039-dd0c-47b4-be42-1edef761c1e6/manager/0.log"
Nov 03 20:09:50.570562 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:50.570536    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-csmwj_c4376e17-400a-4487-bdff-ef9878f40a0e/kuadrant-console-plugin/0.log"
Nov 03 20:09:52.002696 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:52.002633    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2prbp_f7abe1b7-d7bc-402e-a762-e35a95520eb3/discovery/0.log"
Nov 03 20:09:52.377047 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:52.377007    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b4fd54d5-wn7tv_3e5c9e63-14b4-49b8-b8be-44639d2e8d9e/kube-auth-proxy/0.log"
Nov 03 20:09:52.744816 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:52.744718    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-bqzwl_f5da6be6-a4c0-40f5-8f07-d3de1d188394/istio-proxy/0.log"
Nov 03 20:09:53.128242 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:53.128210    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-f5xfb_98ec8c1c-49db-492e-958e-fabe09d044fb/istio-proxy/0.log"
Nov 03 20:09:54.078170 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:54.078135    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-t9ft8_c8d65667-e957-4334-a43a-71d03f1d1000/istio-proxy/0.log"
Nov 03 20:09:56.664130 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:09:56.664096    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:13.254593 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:10:13.254547    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:10:31.891410 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:10:31.891364    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:34.241298 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:10:34.241240    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:35.013708 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:10:35.013674    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-694f8bb88d-h7fff_9a72f77e-2af9-4e75-bbfe-a20447358ef5/manager/0.log"
Nov 03 20:10:35.269610 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:10:35.269505    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-694f8bb88d-h7fff_9a72f77e-2af9-4e75-bbfe-a20447358ef5/manager/0.log"
Nov 03 20:10:43.254761 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:10:43.254705    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:11:13.253803 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:11:13.253759    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:11:22.118739 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:22.118680    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-694f8bb88d-h7fff_9a72f77e-2af9-4e75-bbfe-a20447358ef5/manager/0.log"
Nov 03 20:11:22.397292 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:22.397204    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-694f8bb88d-h7fff_9a72f77e-2af9-4e75-bbfe-a20447358ef5/manager/0.log"
Nov 03 20:11:23.641415 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:23.641375    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:23.785732 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:23.785687    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:26.747022 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:26.746995    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lqnwg_7b22e16a-511b-4d70-a39c-16160ef790b1/global-pull-secret-syncer/0.log"
Nov 03 20:11:27.083272 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:27.083225    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log"
Nov 03 20:11:27.231324 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:27.231277    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-29.us-east-2.compute.internal_dd9e417c99cd1e2d8fd47e9d585da591/haproxy/0.log"
Nov 03 20:11:31.957056 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:31.957018    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:32.240222 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:32.240129    5237 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52ft7_9095ae71-1326-4ec3-9755-164738a4da17/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:33.437767 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:33.437732    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-m8n7h_e2402039-dd0c-47b4-be42-1edef761c1e6/manager/0.log"
Nov 03 20:11:33.497617 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:33.497591    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-csmwj_c4376e17-400a-4487-bdff-ef9878f40a0e/kuadrant-console-plugin/0.log"
Nov 03 20:11:34.910895 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:34.910859    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b945ca1-7ff9-4062-aede-430a2061c9db/alertmanager/0.log"
Nov 03 20:11:34.932034 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:34.932006    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b945ca1-7ff9-4062-aede-430a2061c9db/config-reloader/0.log"
Nov 03 20:11:34.959231 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:34.959201    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b945ca1-7ff9-4062-aede-430a2061c9db/kube-rbac-proxy-web/0.log"
Nov 03 20:11:34.990555 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:34.990524    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b945ca1-7ff9-4062-aede-430a2061c9db/kube-rbac-proxy/0.log"
Nov 03 20:11:35.018848 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:35.018817    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b945ca1-7ff9-4062-aede-430a2061c9db/kube-rbac-proxy-metric/0.log"
Nov 03 20:11:35.047575 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:35.047546    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b945ca1-7ff9-4062-aede-430a2061c9db/prom-label-proxy/0.log"
Nov 03 20:11:35.077966 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:35.077940    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b945ca1-7ff9-4062-aede-430a2061c9db/init-config-reloader/0.log"
Nov 03 20:11:35.262151 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:35.262075    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7d8d7db6bc-592nt_f9b95482-e639-4d23-97a8-59941d9ecbe4/metrics-server/0.log"
Nov 03 20:11:35.527286 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:35.527197    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z24qr_65df60f4-e551-4351-a403-26305bece450/node-exporter/0.log"
Nov 03 20:11:35.559212 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:35.559177    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z24qr_65df60f4-e551-4351-a403-26305bece450/kube-rbac-proxy/0.log"
Nov 03 20:11:35.588860 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:35.588817    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z24qr_65df60f4-e551-4351-a403-26305bece450/init-textfile/0.log"
Nov 03 20:11:36.046914 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:36.046884    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c5449fd55-vnf6b_e5ed038d-cacd-4ec2-b755-d8e714aef422/telemeter-client/0.log"
Nov 03 20:11:36.074161 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:36.074129    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c5449fd55-vnf6b_e5ed038d-cacd-4ec2-b755-d8e714aef422/reload/0.log"
Nov 03 20:11:36.106302 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:36.106273    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c5449fd55-vnf6b_e5ed038d-cacd-4ec2-b755-d8e714aef422/kube-rbac-proxy/0.log"
Nov 03 20:11:36.786459 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:36.786424    5237 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"]
Nov 03 20:11:36.786792 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:36.786777    5237 cpu_manager.go:414] "containerMap: removing container" podUID="ac9dbdbd-6b70-4422-b715-2cc79394ff20" containerName="maas-api"
Nov 03 20:11:36.786792 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:36.786790    5237 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac9dbdbd-6b70-4422-b715-2cc79394ff20" containerName="maas-api"
Nov 03 20:11:36.786934 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:36.786879    5237 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac9dbdbd-6b70-4422-b715-2cc79394ff20" containerName="maas-api"
Nov 03 20:11:36.789784 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:36.789765    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:36.801494 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:36.801466    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"]
Nov 03 20:11:36.887388 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:36.887351    5237 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b8lpv\"/\"default-dockercfg-tjpxw\""
Nov 03 20:11:36.887388 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:36.887367    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b8lpv\"/\"openshift-service-ca.crt\""
Nov 03 20:11:36.887599 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:36.887490    5237 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b8lpv\"/\"kube-root-ca.crt\""
Nov 03 20:11:36.924001 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:36.923969    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22z45\" (UniqueName: \"kubernetes.io/projected/d41e2d6f-17ea-4213-956d-8e5e14f8e24e-kube-api-access-22z45\") pod \"perf-node-gather-daemonset-pnth7\" (UID: \"d41e2d6f-17ea-4213-956d-8e5e14f8e24e\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:36.924190 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:36.924004    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d41e2d6f-17ea-4213-956d-8e5e14f8e24e-podres\") pod \"perf-node-gather-daemonset-pnth7\" (UID: \"d41e2d6f-17ea-4213-956d-8e5e14f8e24e\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:36.924190 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:36.924030    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d41e2d6f-17ea-4213-956d-8e5e14f8e24e-sys\") pod \"perf-node-gather-daemonset-pnth7\" (UID: \"d41e2d6f-17ea-4213-956d-8e5e14f8e24e\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:36.924190 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:36.924153    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d41e2d6f-17ea-4213-956d-8e5e14f8e24e-proc\") pod \"perf-node-gather-daemonset-pnth7\" (UID: \"d41e2d6f-17ea-4213-956d-8e5e14f8e24e\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:36.924349 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:36.924189    5237 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d41e2d6f-17ea-4213-956d-8e5e14f8e24e-lib-modules\") pod \"perf-node-gather-daemonset-pnth7\" (UID: \"d41e2d6f-17ea-4213-956d-8e5e14f8e24e\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:37.025300 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:37.025269    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d41e2d6f-17ea-4213-956d-8e5e14f8e24e-proc\") pod \"perf-node-gather-daemonset-pnth7\" (UID: \"d41e2d6f-17ea-4213-956d-8e5e14f8e24e\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:37.025501 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:37.025317    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d41e2d6f-17ea-4213-956d-8e5e14f8e24e-lib-modules\") pod \"perf-node-gather-daemonset-pnth7\" (UID: \"d41e2d6f-17ea-4213-956d-8e5e14f8e24e\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:37.025501 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:37.025396    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d41e2d6f-17ea-4213-956d-8e5e14f8e24e-proc\") pod \"perf-node-gather-daemonset-pnth7\" (UID: \"d41e2d6f-17ea-4213-956d-8e5e14f8e24e\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:37.025501 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:37.025402    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22z45\" (UniqueName: \"kubernetes.io/projected/d41e2d6f-17ea-4213-956d-8e5e14f8e24e-kube-api-access-22z45\") pod \"perf-node-gather-daemonset-pnth7\" (UID: \"d41e2d6f-17ea-4213-956d-8e5e14f8e24e\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:37.025501 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:37.025450    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d41e2d6f-17ea-4213-956d-8e5e14f8e24e-podres\") pod \"perf-node-gather-daemonset-pnth7\" (UID: \"d41e2d6f-17ea-4213-956d-8e5e14f8e24e\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:37.025501 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:37.025472    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d41e2d6f-17ea-4213-956d-8e5e14f8e24e-lib-modules\") pod \"perf-node-gather-daemonset-pnth7\" (UID: \"d41e2d6f-17ea-4213-956d-8e5e14f8e24e\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:37.025748 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:37.025517    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d41e2d6f-17ea-4213-956d-8e5e14f8e24e-podres\") pod \"perf-node-gather-daemonset-pnth7\" (UID: \"d41e2d6f-17ea-4213-956d-8e5e14f8e24e\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:37.025748 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:37.025483    5237 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d41e2d6f-17ea-4213-956d-8e5e14f8e24e-sys\") pod \"perf-node-gather-daemonset-pnth7\" (UID: \"d41e2d6f-17ea-4213-956d-8e5e14f8e24e\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:37.025748 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:37.025537    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d41e2d6f-17ea-4213-956d-8e5e14f8e24e-sys\") pod \"perf-node-gather-daemonset-pnth7\" (UID: \"d41e2d6f-17ea-4213-956d-8e5e14f8e24e\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:37.039412 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:37.039350    5237 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22z45\" (UniqueName: \"kubernetes.io/projected/d41e2d6f-17ea-4213-956d-8e5e14f8e24e-kube-api-access-22z45\") pod \"perf-node-gather-daemonset-pnth7\" (UID: \"d41e2d6f-17ea-4213-956d-8e5e14f8e24e\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:37.103334 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:37.103298    5237 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:37.375332 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:37.375231    5237 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"]
Nov 03 20:11:37.381295 ip-10-0-133-29 kubenswrapper[5237]: W1103 20:11:37.381265    5237 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd41e2d6f_17ea_4213_956d_8e5e14f8e24e.slice/crio-5c25952313c74b611936cbd5aff0fcc6d9f77bcbd2fc582cd721fb433f404cfe WatchSource:0}: Error finding container 5c25952313c74b611936cbd5aff0fcc6d9f77bcbd2fc582cd721fb433f404cfe: Status 404 returned error can't find the container with id 5c25952313c74b611936cbd5aff0fcc6d9f77bcbd2fc582cd721fb433f404cfe
Nov 03 20:11:37.915764 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:37.915731    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7" event={"ID":"d41e2d6f-17ea-4213-956d-8e5e14f8e24e","Type":"ContainerStarted","Data":"c5ca3c826ec64d47792bfe10bae7713255c8b74a98c5e291bdc25c21298bf1ae"}
Nov 03 20:11:37.915971 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:37.915804    5237 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7" event={"ID":"d41e2d6f-17ea-4213-956d-8e5e14f8e24e","Type":"ContainerStarted","Data":"5c25952313c74b611936cbd5aff0fcc6d9f77bcbd2fc582cd721fb433f404cfe"}
Nov 03 20:11:37.915971 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:37.915869    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:37.935412 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:37.935343    5237 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7" podStartSLOduration=1.935329289 podStartE2EDuration="1.935329289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:37.93450008 +0000 UTC m=+2654.188807297" watchObservedRunningTime="2026-11-03 20:11:37.935329289 +0000 UTC m=+2654.189636539"
Nov 03 20:11:40.687448 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:40.687418    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h6nf4_4ea78abe-c045-41df-b4c8-d5275dcb19e4/dns/0.log"
Nov 03 20:11:40.714722 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:40.714692    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h6nf4_4ea78abe-c045-41df-b4c8-d5275dcb19e4/kube-rbac-proxy/0.log"
Nov 03 20:11:40.833392 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:40.833363    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r2j57_74bbf640-fdad-4295-8810-f471ee4bb4da/dns-node-resolver/0.log"
Nov 03 20:11:41.601490 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:41.601459    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2n8jc_90836408-13df-4995-acec-b7bf40eeec84/node-ca/0.log"
Nov 03 20:11:43.263092 ip-10-0-133-29 kubenswrapper[5237]: E1103 20:11:43.263052    5237 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkjp4" containerName="extractor"
Nov 03 20:11:43.932480 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:43.932450    5237 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-pnth7"
Nov 03 20:11:44.867791 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:44.867755    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2prbp_f7abe1b7-d7bc-402e-a762-e35a95520eb3/discovery/0.log"
Nov 03 20:11:45.339379 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:45.339348    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b4fd54d5-wn7tv_3e5c9e63-14b4-49b8-b8be-44639d2e8d9e/kube-auth-proxy/0.log"
Nov 03 20:11:45.413407 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:45.413377    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-bqzwl_f5da6be6-a4c0-40f5-8f07-d3de1d188394/istio-proxy/0.log"
Nov 03 20:11:45.486984 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:45.486948    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-f5xfb_98ec8c1c-49db-492e-958e-fabe09d044fb/istio-proxy/0.log"
Nov 03 20:11:45.684750 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:45.684659    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-t9ft8_c8d65667-e957-4334-a43a-71d03f1d1000/istio-proxy/0.log"
Nov 03 20:11:47.857066 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:47.857034    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xszjb_24476d57-ccbb-45a7-a594-5ba46b48ea41/serve-healthcheck-canary/0.log"
Nov 03 20:11:48.495424 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:48.495393    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hkjp4_2f6a7918-e7d4-495f-9514-82e883318c90/kube-rbac-proxy/0.log"
Nov 03 20:11:48.527613 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:48.527580    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hkjp4_2f6a7918-e7d4-495f-9514-82e883318c90/exporter/0.log"
Nov 03 20:11:48.558897 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:48.558865    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hkjp4_2f6a7918-e7d4-495f-9514-82e883318c90/extractor/0.log"
Nov 03 20:11:49.232889 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:49.232854    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2prbp_f7abe1b7-d7bc-402e-a762-e35a95520eb3/discovery/0.log"
Nov 03 20:11:49.266922 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:49.266889    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b4fd54d5-wn7tv_3e5c9e63-14b4-49b8-b8be-44639d2e8d9e/kube-auth-proxy/0.log"
Nov 03 20:11:49.305200 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:49.305167    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-bqzwl_f5da6be6-a4c0-40f5-8f07-d3de1d188394/istio-proxy/0.log"
Nov 03 20:11:49.352511 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:49.352473    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-f5xfb_98ec8c1c-49db-492e-958e-fabe09d044fb/istio-proxy/0.log"
Nov 03 20:11:49.452762 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:49.452732    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-t9ft8_c8d65667-e957-4334-a43a-71d03f1d1000/istio-proxy/0.log"
Nov 03 20:11:51.369249 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:51.369217    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7646b49899-sdczk_c577ed1e-9a43-4837-9f01-64bbc37a71b5/manager/0.log"
Nov 03 20:11:51.388781 ip-10-0-133-29 kubenswrapper[5237]: I1103 20:11:51.388755    5237 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7646b49899-sdczk_c577ed1e-9a43-4837-9f01-64bbc37a71b5/copy-manifests/0.log"