Oct 30 21:50:10.739580 ip-10-0-130-162 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 21:50:10.739589 ip-10-0-130-162 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 21:50:10.739596 ip-10-0-130-162 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 21:50:10.739838 ip-10-0-130-162 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 21:50:20.988847 ip-10-0-130-162 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 21:50:20.988864 ip-10-0-130-162 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a9417424cc8a4cca944852839719960a --
Oct 30 21:52:20.235247 ip-10-0-130-162 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:52:20.643899 ip-10-0-130-162 kubenswrapper[2539]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:52:20.643899 ip-10-0-130-162 kubenswrapper[2539]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:52:20.643899 ip-10-0-130-162 kubenswrapper[2539]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:52:20.643899 ip-10-0-130-162 kubenswrapper[2539]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:52:20.649829 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649719    2539 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:52:20.649829 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649829    2539 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:52:20.649829 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649833    2539 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649836    2539 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649839    2539 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649842    2539 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649844    2539 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649848    2539 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649854    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649857    2539 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649860    2539 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649863    2539 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649866    2539 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649868    2539 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649871    2539 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649873    2539 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649876    2539 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649879    2539 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649882    2539 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649884    2539 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649887    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649890    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:52:20.649926 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649892    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649895    2539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649897    2539 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649900    2539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649902    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649905    2539 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649908    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649910    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649913    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649915    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649918    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649920    2539 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649923    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649925    2539 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649929    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649932    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649935    2539 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649938    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649941    2539 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:52:20.650414 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649944    2539 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649963    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649968    2539 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649971    2539 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649973    2539 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649976    2539 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649978    2539 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649981    2539 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649984    2539 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649986    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649989    2539 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649991    2539 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649994    2539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649997    2539 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.649999    2539 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650001    2539 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650004    2539 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650007    2539 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650009    2539 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650012    2539 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:52:20.650847 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650014    2539 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650017    2539 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650019    2539 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650022    2539 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650024    2539 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650027    2539 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650030    2539 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650035    2539 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650039    2539 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650042    2539 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650045    2539 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650048    2539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650050    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650053    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650056    2539 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650058    2539 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650061    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650063    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650065    2539 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:52:20.651328 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650068    2539 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650071    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650074    2539 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650076    2539 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650079    2539 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650081    2539 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650085    2539 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650089    2539 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650092    2539 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650094    2539 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650097    2539 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650099    2539 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650102    2539 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650104    2539 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650107    2539 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650109    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650112    2539 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650114    2539 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650116    2539 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650119    2539 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:52:20.651909 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650123    2539 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650126    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650129    2539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650132    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650134    2539 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650137    2539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650943    2539 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650970    2539 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650974    2539 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650979    2539 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650983    2539 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650986    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650988    2539 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650991    2539 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650994    2539 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650996    2539 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.650999    2539 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651002    2539 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651004    2539 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:52:20.652731 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651009    2539 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651014    2539 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651017    2539 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651020    2539 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651022    2539 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651025    2539 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651028    2539 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651030    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651033    2539 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651036    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651038    2539 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651041    2539 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651043    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651046    2539 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651049    2539 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651052    2539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651054    2539 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651057    2539 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651060    2539 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651062    2539 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:52:20.653215 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651065    2539 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651075    2539 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651078    2539 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651081    2539 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651083    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651086    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651089    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651091    2539 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651094    2539 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651096    2539 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651099    2539 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651102    2539 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651104    2539 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651107    2539 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651109    2539 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651112    2539 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651114    2539 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651117    2539 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651119    2539 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651122    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:52:20.653690 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651126    2539 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651129    2539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651132    2539 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651135    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651137    2539 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651141    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651143    2539 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651146    2539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651148    2539 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651151    2539 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651154    2539 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651156    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651159    2539 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651162    2539 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651165    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651168    2539 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651170    2539 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651173    2539 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651176    2539 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651178    2539 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:52:20.654190 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651180    2539 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651183    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651186    2539 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651188    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651191    2539 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651193    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651196    2539 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651198    2539 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651201    2539 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651203    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651206    2539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651208    2539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651211    2539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651213    2539 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651215    2539 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651218    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651220    2539 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651223    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651226    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:52:20.654673 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651228    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651230    2539 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651233    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651236    2539 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651239    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651242    2539 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651244    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651247    2539 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651250    2539 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651252    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651255    2539 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651257    2539 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651260    2539 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.651262    2539 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653211    2539 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653224    2539 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653227    2539 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653230    2539 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653234    2539 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:52:20.655167 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653236    2539 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653239    2539 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653242    2539 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653244    2539 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653247    2539 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653251    2539 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653255    2539 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653258    2539 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653261    2539 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653263    2539 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653266    2539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653269    2539 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653271    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653274    2539 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653276    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653279    2539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653281    2539 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653284    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653287    2539 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:52:20.655620 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653289    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653292    2539 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653295    2539 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653298    2539 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653301    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653306    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653309    2539 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653312    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653315    2539 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653318    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653320    2539 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653323    2539 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653325    2539 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653328    2539 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653330    2539 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653333    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653335    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653338    2539 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653341    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653343    2539 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:52:20.656082 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653346    2539 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653350    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653352    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653354    2539 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653357    2539 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653359    2539 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653363    2539 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653366    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653368    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653371    2539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653373    2539 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653376    2539 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653378    2539 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653381    2539 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653383    2539 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653386    2539 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653388    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653391    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653393    2539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653395    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:52:20.656555 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653398    2539 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653402    2539 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653404    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653407    2539 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653409    2539 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653412    2539 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653415    2539 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653417    2539 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653420    2539 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653422    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653425    2539 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653428    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653430    2539 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653433    2539 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653435    2539 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653438    2539 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653440    2539 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653442    2539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653445    2539 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:52:20.657050 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653448    2539 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653451    2539 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653453    2539 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653456    2539 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653458    2539 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653461    2539 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653465    2539 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653469    2539 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653472    2539 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653475    2539 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653478    2539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653480    2539 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653483    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653486    2539 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653488    2539 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653491    2539 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653494    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653496    2539 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653499    2539 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:52:20.657502 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653501    2539 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653504    2539 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653506    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.653508    2539 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654481    2539 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654491    2539 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654498    2539 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654503    2539 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654508    2539 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654511    2539 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654515    2539 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654519    2539 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654523    2539 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654526    2539 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654531    2539 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654534    2539 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654537    2539 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654540    2539 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654543    2539 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654546    2539 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654549    2539 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654552    2539 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654557    2539 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:52:20.657962 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654560    2539 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654563    2539 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654566    2539 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654570    2539 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654574    2539 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654577    2539 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654580    2539 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654584    2539 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654587    2539 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654590    2539 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654593    2539 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654596    2539 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654599    2539 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654604    2539 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654607    2539 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654610    2539 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654612    2539 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654615    2539 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654618    2539 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654623    2539 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654626    2539 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654629    2539 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654633    2539 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654636    2539 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:52:20.658527 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654641    2539 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654644    2539 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654647    2539 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654650    2539 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654653    2539 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654656    2539 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654660    2539 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654663    2539 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654665    2539 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654669    2539 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654671    2539 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654676    2539 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654680    2539 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654683    2539 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654686    2539 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654689    2539 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654693    2539 flags.go:64] FLAG: --help="false"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654695    2539 flags.go:64] FLAG: --hostname-override="ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654699    2539 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654703    2539 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654706    2539 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654709    2539 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654712    2539 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:52:20.659155 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654715    2539 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654718    2539 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654721    2539 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654724    2539 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654727    2539 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654730    2539 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654736    2539 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654739    2539 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654742    2539 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654745    2539 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654749    2539 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654752    2539 flags.go:64] FLAG: --lock-file=""
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654755    2539 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654758    2539 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654761    2539 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654770    2539 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654773    2539 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654777    2539 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654780    2539 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654783    2539 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654787    2539 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654790    2539 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654792    2539 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654797    2539 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654800    2539 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:52:20.659684 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654804    2539 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654807    2539 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654810    2539 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654813    2539 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654815    2539 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654818    2539 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654821    2539 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654824    2539 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654831    2539 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654834    2539 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654837    2539 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654840    2539 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654843    2539 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654847    2539 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654850    2539 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654853    2539 flags.go:64] FLAG: --port="10250"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654856    2539 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654859    2539 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0fdf70df224af0e3b"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654863    2539 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654866    2539 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654869    2539 flags.go:64] FLAG: --register-node="true"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654872    2539 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654876    2539 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654879    2539 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654881    2539 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654884    2539 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:52:20.660281 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654888    2539 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654891    2539 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654894    2539 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654897    2539 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654900    2539 flags.go:64] FLAG: --runonce="false"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654903    2539 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654906    2539 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654909    2539 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654912    2539 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654915    2539 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654918    2539 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654921    2539 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654924    2539 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654927    2539 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654930    2539 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654933    2539 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654935    2539 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654938    2539 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654941    2539 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654945    2539 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654966    2539 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654969    2539 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654972    2539 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654976    2539 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654979    2539 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:52:20.660890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654983    2539 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654986    2539 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654989    2539 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654991    2539 flags.go:64] FLAG: --v="2"
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.654996    2539 flags.go:64] FLAG: --version="false"
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.655000    2539 flags.go:64] FLAG: --vmodule=""
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.655004    2539 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.655007    2539 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655141    2539 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655145    2539 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655148    2539 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655151    2539 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655154    2539 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655156    2539 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655158    2539 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655161    2539 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655164    2539 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655166    2539 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655169    2539 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655172    2539 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655174    2539 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:52:20.661542 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655177    2539 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655180    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655182    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655185    2539 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655188    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655192    2539 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655195    2539 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655197    2539 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655199    2539 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655202    2539 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655204    2539 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655207    2539 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655210    2539 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655212    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655215    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655218    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655220    2539 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655223    2539 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655225    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655230    2539 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:52:20.662079 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655232    2539 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655236    2539 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655241    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655244    2539 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655247    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655249    2539 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655252    2539 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655254    2539 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655257    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655260    2539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655262    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655265    2539 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655267    2539 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655270    2539 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655273    2539 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655275    2539 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655278    2539 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655281    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655284    2539 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:52:20.662558 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655286    2539 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655289    2539 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655291    2539 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655294    2539 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655296    2539 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655299    2539 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655302    2539 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655305    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655307    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655310    2539 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655312    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655315    2539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655319    2539 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655323    2539 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655327    2539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655330    2539 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655333    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655335    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655338    2539 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:52:20.663903 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655340    2539 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655343    2539 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655345    2539 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655348    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655350    2539 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655353    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655356    2539 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655358    2539 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655361    2539 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655363    2539 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655366    2539 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655370    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655372    2539 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655375    2539 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655377    2539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655380    2539 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655382    2539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655385    2539 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655387    2539 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:52:20.664377 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655390    2539 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:52:20.664850 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655392    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:52:20.664850 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655395    2539 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:52:20.664850 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655398    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:52:20.664850 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655400    2539 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:52:20.664850 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655403    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:52:20.664850 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655407    2539 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:52:20.664850 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655410    2539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:52:20.664850 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655412    2539 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:52:20.664850 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655415    2539 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:52:20.664850 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655418    2539 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:52:20.664850 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655420    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:52:20.664850 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655423    2539 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:52:20.664850 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655425    2539 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:52:20.664850 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655428    2539 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:52:20.664850 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655430    2539 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:52:20.664850 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.655456    2539 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655491    2539 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655496    2539 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655499    2539 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655502    2539 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655505    2539 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655507    2539 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655510    2539 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655514    2539 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655517    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655519    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655522    2539 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655524    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655527    2539 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655529    2539 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655531    2539 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655534    2539 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655537    2539 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655540    2539 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655542    2539 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:52:20.665327 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655545    2539 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655547    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655550    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655554    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655556    2539 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655559    2539 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655561    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655564    2539 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655566    2539 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655570    2539 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655574    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655577    2539 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655579    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655582    2539 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655584    2539 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655587    2539 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655590    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655592    2539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655595    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:52:20.665787 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655597    2539 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655601    2539 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655604    2539 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655606    2539 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655609    2539 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655611    2539 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655614    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655616    2539 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655619    2539 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655621    2539 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655624    2539 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655627    2539 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655631    2539 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655634    2539 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655636    2539 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655639    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655642    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655645    2539 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655648    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655650    2539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:52:20.666250 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655653    2539 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655657    2539 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655661    2539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655665    2539 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655668    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655670    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655673    2539 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655675    2539 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655678    2539 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655681    2539 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655683    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655686    2539 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655688    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655692    2539 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655695    2539 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655698    2539 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655700    2539 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655703    2539 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655705    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:52:20.666754 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655708    2539 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655710    2539 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655713    2539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655715    2539 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655718    2539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655720    2539 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655723    2539 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655726    2539 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655728    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655732    2539 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655734    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655737    2539 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655740    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655742    2539 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655745    2539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655747    2539 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655750    2539 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655754    2539 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655756    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:52:20.667220 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655759    2539 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655761    2539 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655764    2539 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655766    2539 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655769    2539 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655771    2539 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655774    2539 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655777    2539 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655780    2539 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.655782    2539 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.662847    2539 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.662863    2539 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.662880    2539 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.662885    2539 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.662941    2539 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.662962    2539 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.662968    2539 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.662972    2539 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.662975    2539 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.662977    2539 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:52:20.667676 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.662980    2539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.662983    2539 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.662986    2539 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.662988    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.662991    2539 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.662994    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.662996    2539 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.662999    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663002    2539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663004    2539 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663007    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663009    2539 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663012    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663015    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663019    2539 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663022    2539 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663025    2539 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663027    2539 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663030    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:52:20.668162 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663032    2539 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663035    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663037    2539 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663039    2539 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663043    2539 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663045    2539 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663048    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663050    2539 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663053    2539 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663055    2539 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663058    2539 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663060    2539 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663063    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663065    2539 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663068    2539 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663070    2539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663073    2539 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663075    2539 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663077    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663080    2539 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:52:20.668596 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663082    2539 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663085    2539 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663087    2539 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663090    2539 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663092    2539 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663095    2539 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663098    2539 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663100    2539 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663103    2539 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663106    2539 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663108    2539 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663111    2539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663114    2539 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663116    2539 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663119    2539 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663123    2539 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663127    2539 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663131    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663134    2539 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663137    2539 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:52:20.669108 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663139    2539 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663142    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663144    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663147    2539 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663149    2539 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663152    2539 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663154    2539 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663157    2539 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663159    2539 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663162    2539 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663164    2539 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663167    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663169    2539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663171    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663174    2539 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663176    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663179    2539 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663181    2539 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663184    2539 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663187    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:52:20.669603 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663190    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663192    2539 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663195    2539 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663197    2539 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663200    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663203    2539 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663205    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663208    2539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663211    2539 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663213    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663217    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663220    2539 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663222    2539 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663225    2539 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663227    2539 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663230    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663232    2539 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663235    2539 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663238    2539 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:52:20.670130 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663242    2539 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:52:20.670563 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663246    2539 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:52:20.670563 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.663272    2539 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:52:20.670563 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663420    2539 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:52:20.670563 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663424    2539 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:52:20.670563 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663427    2539 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:52:20.670563 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663430    2539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:52:20.670563 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663433    2539 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:52:20.670563 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663435    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:52:20.670563 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663438    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:52:20.670563 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663440    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:52:20.670563 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663443    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:52:20.670563 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663446    2539 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:52:20.670563 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663449    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:52:20.670563 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663452    2539 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:52:20.670563 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663454    2539 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:52:20.670563 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663457    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663459    2539 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663462    2539 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663465    2539 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663467    2539 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663470    2539 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663472    2539 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663475    2539 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663478    2539 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663480    2539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663483    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663485    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663488    2539 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663490    2539 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663493    2539 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663495    2539 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663498    2539 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663500    2539 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663502    2539 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663505    2539 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:52:20.670933 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663508    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663510    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663513    2539 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663515    2539 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663518    2539 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663520    2539 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663523    2539 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663525    2539 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663528    2539 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663531    2539 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663533    2539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663536    2539 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663538    2539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663541    2539 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663543    2539 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663546    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663548    2539 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663550    2539 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663553    2539 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663555    2539 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:52:20.671430 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663558    2539 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663561    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663563    2539 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663565    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663568    2539 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663571    2539 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663573    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663576    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663578    2539 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663581    2539 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663583    2539 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663585    2539 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663588    2539 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663590    2539 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663593    2539 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663595    2539 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663598    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663600    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663602    2539 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:52:20.671970 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663605    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663608    2539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663611    2539 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663613    2539 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663616    2539 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663618    2539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663622    2539 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663626    2539 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663629    2539 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663631    2539 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663635    2539 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663638    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663641    2539 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663643    2539 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663645    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663649    2539 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663651    2539 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663654    2539 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663656    2539 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:52:20.672425 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663659    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:52:20.672900 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663661    2539 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:52:20.672900 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663664    2539 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:52:20.672900 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663668    2539 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:52:20.672900 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663672    2539 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:52:20.672900 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663674    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:52:20.672900 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663677    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:52:20.672900 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663680    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:52:20.672900 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663682    2539 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:52:20.672900 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663685    2539 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:52:20.672900 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663687    2539 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:52:20.672900 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663690    2539 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:52:20.672900 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663692    2539 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:52:20.672900 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663695    2539 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:52:20.672900 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663697    2539 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:52:20.672900 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.663721    2539 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663754    2539 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663758    2539 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663761    2539 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663764    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663767    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663769    2539 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663772    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663775    2539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663777    2539 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663780    2539 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663783    2539 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663785    2539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663803    2539 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663808    2539 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663811    2539 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663813    2539 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663816    2539 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663818    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:52:20.673349 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663821    2539 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663823    2539 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663826    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663828    2539 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663831    2539 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663833    2539 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663836    2539 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663838    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663841    2539 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663843    2539 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663847    2539 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663851    2539 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663855    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663858    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663861    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663864    2539 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663867    2539 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663869    2539 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663872    2539 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:52:20.673786 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663874    2539 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663877    2539 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663879    2539 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663882    2539 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663884    2539 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663887    2539 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663889    2539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663892    2539 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663894    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663897    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663900    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663902    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663905    2539 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663907    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663910    2539 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663913    2539 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663915    2539 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663918    2539 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663920    2539 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663923    2539 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:52:20.674358 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663925    2539 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663928    2539 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663930    2539 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663933    2539 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663935    2539 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663938    2539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663940    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663943    2539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663945    2539 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663965    2539 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663968    2539 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663970    2539 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663973    2539 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663975    2539 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663978    2539 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663980    2539 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663983    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663985    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663988    2539 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663992    2539 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:52:20.674815 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663994    2539 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.663997    2539 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664002    2539 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664004    2539 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664007    2539 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664009    2539 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664012    2539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664014    2539 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664016    2539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664019    2539 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664021    2539 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664024    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664026    2539 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664028    2539 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664031    2539 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664033    2539 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664036    2539 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664038    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664042    2539 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664045    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:52:20.675309 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664047    2539 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:52:20.675774 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664050    2539 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:52:20.675774 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664052    2539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:52:20.675774 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664054    2539 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:52:20.675774 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664057    2539 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:52:20.675774 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664059    2539 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:52:20.675774 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664062    2539 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:52:20.675774 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664064    2539 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:52:20.675774 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:20.664067    2539 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:52:20.675774 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.664128    2539 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:52:20.675774 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.666484    2539 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 21:52:20.675774 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.667379    2539 server.go:1014] "Starting client certificate rotation"
Oct 30 21:52:20.675774 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.667473    2539 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:52:20.675774 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.667506    2539 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:52:20.675774 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.669469    2539 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:52:20.675774 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.671273    2539 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:52:20.681643 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.681627    2539 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:52:20.685095 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.685080    2539 log.go:25] "Validated CRI v1 image API"
Oct 30 21:52:20.685880 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.685854    2539 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:52:20.691066 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.691045    2539 fs.go:135] Filesystem UUIDs: map[029e9b58-7870-4ae1-a993-6b376facb40f:/dev/nvme0n1p3 351ae0e9-eae6-4b38-868b-f7742a75030b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 21:52:20.691122 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.691064    2539 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 21:52:20.696271 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.696168    2539 manager.go:217] Machine: {Timestamp:2026-10-30 21:52:20.694692413 +0000 UTC m=+0.336582010 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3084767 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c85a065da967d5d042e836395660c SystemUUID:ec2c85a0-65da-967d-5d04-2e836395660c BootID:a9417424-cc8a-4cca-9448-52839719960a Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:48:5d:4d:79:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:48:5d:4d:79:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:12:98:a5:97:af Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:52:20.696271 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.696274    2539 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:52:20.696381 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.696348    2539 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:52:20.699319 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.699297    2539 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:52:20.699452 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.699321    2539 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-162.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:52:20.699492 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.699456    2539 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:52:20.699492 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.699463    2539 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:52:20.699546 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.699496    2539 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:52:20.699546 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.699514    2539 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:52:20.700696 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.700686    2539 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:52:20.704785 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.704773    2539 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:52:20.704894 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.704886    2539 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:52:20.705698 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.705688    2539 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:52:20.705752 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.705702    2539 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:52:20.705752 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.705713    2539 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:52:20.705752 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.705722    2539 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:52:20.705752 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.705731    2539 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:52:20.706075 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.706064    2539 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:52:20.709220 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.709201    2539 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:52:20.716052 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.716038    2539 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:52:20.717060 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.717048    2539 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:52:20.717103 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.717078    2539 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:52:20.717227 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.717217    2539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:52:20.717270 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.717233    2539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:52:20.717270 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.717238    2539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:52:20.717270 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.717245    2539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:52:20.717270 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.717250    2539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:52:20.717270 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.717256    2539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:52:20.718064 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.718056    2539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:52:20.718094 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.718066    2539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:52:20.718094 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.718073    2539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:52:20.718094 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.718079    2539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:52:20.718094 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.718087    2539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:52:20.718303 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.718096    2539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:52:20.718817 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.718808    2539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:52:20.718817 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.718818    2539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:52:20.722524 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.722510    2539 server.go:1263] "Started kubelet"
Oct 30 21:52:20.722642 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.722612    2539 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:52:20.723305 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.722610    2539 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:52:20.723385 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.723320    2539 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:52:20.723473 ip-10-0-130-162 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:52:20.725640 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.725622    2539 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:52:20.725907 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.725896    2539 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:52:20.728821 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:20.728797    2539 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 21:52:20.730805 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.730787    2539 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:52:20.731515 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.731495    2539 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:52:20.731862 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:20.731842    2539 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:52:20.732392 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.732371    2539 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:52:20.732461 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.732396    2539 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:52:20.732507 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.732475    2539 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:52:20.732507 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.732484    2539 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:52:20.732620 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.732598    2539 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:52:20.734240 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.734222    2539 factory.go:55] Registering systemd factory
Oct 30 21:52:20.734240 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.734243    2539 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:52:20.734751 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.734733    2539 factory.go:153] Registering CRI-O factory
Oct 30 21:52:20.734816 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.734756    2539 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:52:20.734816 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.734809    2539 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:52:20.734900 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.734830    2539 factory.go:103] Registering Raw factory
Oct 30 21:52:20.734900 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.734845    2539 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:52:20.736318 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.736303    2539 manager.go:319] Starting recovery of all containers
Oct 30 21:52:20.741801 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.741757    2539 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:52:20.746043 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.745915    2539 manager.go:324] Recovery completed
Oct 30 21:52:20.749889 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.749876    2539 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:52:20.751622 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.751603    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:52:20.751693 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.751635    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:52:20.751693 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.751646    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:52:20.752174 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.752160    2539 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:52:20.752174 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.752171    2539 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:52:20.752284 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.752187    2539 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:52:20.752284 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.752207    2539 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:52:20.752885 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.752871    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:52:20.752963 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.752902    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:52:20.752963 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.752917    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:52:20.754342 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.754329    2539 policy_none.go:50] "Start"
Oct 30 21:52:20.754395 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.754345    2539 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:52:20.754395 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.754359    2539 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:52:20.756388 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.756373    2539 policy_none.go:44] "Start"
Oct 30 21:52:20.815681 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.805534    2539 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:52:20.815681 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.806766    2539 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:52:20.815681 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.806787    2539 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:52:20.815681 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.806804    2539 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:52:20.815681 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:20.806835    2539 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:52:20.815681 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.810702    2539 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:52:20.815681 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:20.810752    2539 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:52:20.815681 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.810765    2539 server.go:87] "Starting device plugin registration server"
Oct 30 21:52:20.815681 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.811001    2539 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:52:20.815681 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.811013    2539 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:52:20.815681 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.811099    2539 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:52:20.815681 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.811167    2539 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:52:20.815681 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.811176    2539 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:52:20.815681 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:20.811651    2539 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:52:20.815681 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:20.811689    2539 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:52:20.864465 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.864441    2539 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-162.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 21:52:20.907751 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.907714    2539 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal"]
Oct 30 21:52:20.907879 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.907784    2539 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:52:20.908799 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.908785    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:52:20.908863 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.908814    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:52:20.908863 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.908826    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:52:20.911097 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.911082    2539 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:52:20.911153 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.911091    2539 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:52:20.911202 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.911188    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:20.911240 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.911216    2539 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:52:20.913086 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.913065    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:52:20.913196 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.913098    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:52:20.913196 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.913074    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:52:20.913196 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.913113    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:52:20.913196 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.913132    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:52:20.913196 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:20.913122    2539 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-162.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 21:52:20.913196 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.913148    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:52:20.913196 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.913076    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:52:20.913196 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.913199    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:52:20.913522 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.913214    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:52:20.913522 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.913241    2539 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:20.913627 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:20.912410    2539 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-162.us-west-1.compute.internal.18e36d3299f6fe57  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-162.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-162.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:52:20.722482775 +0000 UTC m=+0.364372371,LastTimestamp:2026-10-30 21:52:20.722482775 +0000 UTC m=+0.364372371,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-162.us-west-1.compute.internal,}"
Oct 30 21:52:20.915284 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.915266    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:20.915362 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.915293    2539 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:52:20.916028 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.916011    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:52:20.916101 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.916041    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:52:20.916101 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.916053    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:52:20.945546 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:20.945528    2539 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found" node="ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:20.948036 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:20.948017    2539 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found" node="ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:20.965313 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:20.965289    2539 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 21:52:20.965472 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:20.965452    2539 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-162.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 21:52:20.965571 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:20.965474    2539 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 21:52:20.965571 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:20.965487    2539 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 30 21:52:20.965713 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:20.965511    2539 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-162.us-west-1.compute.internal.18e36d329bb39aad  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-162.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-162.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-162.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:52:20.751620781 +0000 UTC m=+0.393510376,LastTimestamp:2026-10-30 21:52:20.751620781 +0000 UTC m=+0.393510376,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-162.us-west-1.compute.internal,}"
Oct 30 21:52:20.988710 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.988688    2539 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:52:20.988803 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:20.988756    2539 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lnr24"
Oct 30 21:52:21.022081 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:21.022052    2539 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-130-162.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:21.022184 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:21.022082    2539 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-162.us-west-1.compute.internal.18e36d329bb3e76f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-162.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-162.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-162.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:52:20.751640431 +0000 UTC m=+0.393530026,LastTimestamp:2026-10-30 21:52:20.751640431 +0000 UTC m=+0.393530026,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-162.us-west-1.compute.internal,}"
Oct 30 21:52:21.033195 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.033175    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a78e001a5670be0097c09eab27b6eea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"0a78e001a5670be0097c09eab27b6eea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:21.033279 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.033204    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a78e001a5670be0097c09eab27b6eea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"0a78e001a5670be0097c09eab27b6eea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:21.033279 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.033227    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a49dc1ff1f4db21ad91c65b8f36133e5-config\") pod \"kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"a49dc1ff1f4db21ad91c65b8f36133e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:21.133385 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.133350    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a78e001a5670be0097c09eab27b6eea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"0a78e001a5670be0097c09eab27b6eea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:21.133385 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.133386    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a78e001a5670be0097c09eab27b6eea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"0a78e001a5670be0097c09eab27b6eea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:21.133594 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.133411    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a49dc1ff1f4db21ad91c65b8f36133e5-config\") pod \"kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"a49dc1ff1f4db21ad91c65b8f36133e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:21.133594 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.133457    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a78e001a5670be0097c09eab27b6eea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"0a78e001a5670be0097c09eab27b6eea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:21.133594 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.133472    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a49dc1ff1f4db21ad91c65b8f36133e5-config\") pod \"kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"a49dc1ff1f4db21ad91c65b8f36133e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:21.133594 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.133472    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a78e001a5670be0097c09eab27b6eea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"0a78e001a5670be0097c09eab27b6eea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:21.217072 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:21.217010    2539 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-162.us-west-1.compute.internal\" not found" node="ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:21.223105 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.223087    2539 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:52:21.224264 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.224246    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:52:21.224382 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.224285    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:52:21.224382 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.224299    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:52:21.224382 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.224335    2539 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:21.249264 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.249238    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:21.253267 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.253251    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:21.278677 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.278660    2539 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:21.278742 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:21.278686    2539 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-162.us-west-1.compute.internal\": node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:52:21.396140 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:21.395470    2539 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:52:21.496568 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:21.496472    2539 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:52:21.596995 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:21.596947    2539 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:52:21.667830 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.667804    2539 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 21:52:21.668312 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:21.667990    2539 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a9adc942bda134c9d9d9f5a2e264780f-d8cf5ed739e19940.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-130-162.us-west-1.compute.internal.18e36d329bb39aad\": read tcp 10.0.130.162:33666->77.113.175.168:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-130-162.us-west-1.compute.internal.18e36d329bb39aad  default   5039 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-162.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-162.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-162.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:52:20 +0000 UTC,LastTimestamp:2026-10-30 21:52:20.913085742 +0000 UTC m=+0.554975340,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-162.us-west-1.compute.internal,}"
Oct 30 21:52:21.698076 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:21.698056    2539 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:52:21.798812 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:21.798746    2539 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:52:21.898889 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:21.898861    2539 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:52:21.994522 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.994473    2539 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:47:21 +0000 UTC" deadline="2028-08-15 07:01:26.352000966 +0000 UTC"
Oct 30 21:52:21.994522 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:21.994514    2539 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15705h9m4.357489803s"
Oct 30 21:52:21.999635 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:21.999616    2539 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:52:22.007767 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.007752    2539 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:52:22.009080 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.009062    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:52:22.009167 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.009095    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:52:22.009167 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.009105    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:52:22.100163 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:22.100089    2539 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:52:22.103374 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.103356    2539 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:52:22.200513 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:22.200475    2539 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:52:22.213828 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.213804    2539 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:52:22.301028 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:22.300996    2539 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:52:22.379668 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:22.379635    2539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49dc1ff1f4db21ad91c65b8f36133e5.slice/crio-fab09ade6f257048793a17ee5835c08a606827a22dc71fb0ed9e264cb4c08112 WatchSource:0}: Error finding container fab09ade6f257048793a17ee5835c08a606827a22dc71fb0ed9e264cb4c08112: Status 404 returned error can't find the container with id fab09ade6f257048793a17ee5835c08a606827a22dc71fb0ed9e264cb4c08112
Oct 30 21:52:22.380285 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:22.380263    2539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a78e001a5670be0097c09eab27b6eea.slice/crio-d7ba18eb6873828ad9148d7cbc35d8c0d98f033645b440cb10bd1fe3ed780173 WatchSource:0}: Error finding container d7ba18eb6873828ad9148d7cbc35d8c0d98f033645b440cb10bd1fe3ed780173: Status 404 returned error can't find the container with id d7ba18eb6873828ad9148d7cbc35d8c0d98f033645b440cb10bd1fe3ed780173
Oct 30 21:52:22.384576 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.384561    2539 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:52:22.401513 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:22.401482    2539 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:52:22.473039 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.473017    2539 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:52:22.502094 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:22.502067    2539 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:52:22.506777 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.506757    2539 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:52:22.532808 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.532777    2539 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:22.594510 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.594484    2539 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:52:22.595514 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.595497    2539 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:52:22.654623 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.654603    2539 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:52:22.706248 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.706228    2539 apiserver.go:52] "Watching apiserver"
Oct 30 21:52:22.733475 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.733445    2539 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 21:52:22.812799 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.812746    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal" event={"ID":"a49dc1ff1f4db21ad91c65b8f36133e5","Type":"ContainerStarted","Data":"fab09ade6f257048793a17ee5835c08a606827a22dc71fb0ed9e264cb4c08112"}
Oct 30 21:52:22.813922 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.813892    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal" event={"ID":"0a78e001a5670be0097c09eab27b6eea","Type":"ContainerStarted","Data":"d7ba18eb6873828ad9148d7cbc35d8c0d98f033645b440cb10bd1fe3ed780173"}
Oct 30 21:52:22.816112 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.816097    2539 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:52:22.816372 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.816357    2539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-kprst","openshift-network-diagnostics/network-check-target-b5h24","openshift-ovn-kubernetes/ovnkube-node-l8xr5","kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-7hbsq","openshift-dns/node-resolver-x2gfj","kube-system/konnectivity-agent-2mskp","openshift-network-operator/iptables-alerter-j68lz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb","openshift-multus/multus-additional-cni-plugins-28mbh","openshift-multus/multus-lh5dm","kube-system/global-pull-secret-syncer-4tlxp","openshift-image-registry/node-ca-pcjh8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"]
Oct 30 21:52:22.820438 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.820423    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.822625 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.822608    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:52:22.825207 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.825191    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.828367 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.828348    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:52:22.830673 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.830654    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.833055 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.833029    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:52:22.835494 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.835477    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:22.837834 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.837811    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2mskp"
Oct 30 21:52:22.840159 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.840143    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:22.840227 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:22.840199    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5h24" podUID="60e220ff-82ff-4d25-b53d-6e98dd4da3fd"
Oct 30 21:52:22.842465 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.842448    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysctl-d\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.842545 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.842473    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysctl-conf\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.842545 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.842496    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-systemd\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.842545 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.842528    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:22.842668 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.842543    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-var-lib-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.842668 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.842588    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.842668 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:22.842589    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kprst" podUID="dbf75704-af7b-44f6-bfcb-b2b2e943f5e6"
Oct 30 21:52:22.842668 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.842613    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-cni-netd\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.842866 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.842689    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.842866 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.842730    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-systemd-units\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.842866 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.842770    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d46f22d3-4503-41af-98d9-c641d343de28-ovn-node-metrics-cert\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.842866 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.842802    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-slash\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.842866 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.842840    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-log-socket\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.843106 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.842871    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-modprobe-d\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.843106 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.842907    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysconfig\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.843106 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.842940    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-sys\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.843106 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.842962    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz8q9\" (UniqueName: \"kubernetes.io/projected/78e61e36-d7f2-432e-af30-16a093b57e7f-kube-api-access-hz8q9\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.843106 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.842993    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-serviceca\") pod \"node-ca-pcjh8\" (UID: \"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098\") " pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:52:22.843106 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843021    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-etc-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.843106 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843039    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-run-ovn-kubernetes\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.843106 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843071    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-cni-bin\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.843106 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843086    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-ovnkube-script-lib\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.843106 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843103    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-kubernetes\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.843473 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843134    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-systemd\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.843473 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843149    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-run-netns\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.843473 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843166    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-node-log\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.843473 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843182    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb4lx\" (UniqueName: \"kubernetes.io/projected/d46f22d3-4503-41af-98d9-c641d343de28-kube-api-access-kb4lx\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.843473 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843197    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-lib-modules\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.843473 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843211    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-tuned\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.843473 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843251    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-cnibin\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.843473 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843267    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-cni-binary-copy\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.843473 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843287    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxs96\" (UniqueName: \"kubernetes.io/projected/6aea3c42-943b-4532-b21d-64ada9922ed2-kube-api-access-kxs96\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.843473 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843301    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-host\") pod \"node-ca-pcjh8\" (UID: \"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098\") " pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:52:22.843473 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843316    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-ovnkube-config\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.843473 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843332    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78e61e36-d7f2-432e-af30-16a093b57e7f-tmp\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.843473 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843347    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d63a68-43fa-4e92-9a39-541cf8983a89-tmp-dir\") pod \"node-resolver-x2gfj\" (UID: \"53d63a68-43fa-4e92-9a39-541cf8983a89\") " pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:52:22.843473 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843363    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.843473 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843379    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlk69\" (UniqueName: \"kubernetes.io/projected/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-kube-api-access-vlk69\") pod \"node-ca-pcjh8\" (UID: \"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098\") " pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:52:22.843473 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843394    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-kubelet\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.843985 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843442    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14434183-f8cd-4f6e-aab4-d8516d44ec74-iptables-alerter-script\") pod \"iptables-alerter-j68lz\" (UID: \"14434183-f8cd-4f6e-aab4-d8516d44ec74\") " pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:52:22.843985 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843458    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14434183-f8cd-4f6e-aab4-d8516d44ec74-host-slash\") pod \"iptables-alerter-j68lz\" (UID: \"14434183-f8cd-4f6e-aab4-d8516d44ec74\") " pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:52:22.843985 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843473    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-run\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.843985 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843518    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.843985 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843535    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-var-lib-kubelet\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.843985 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843562    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-host\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.843985 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843606    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53d63a68-43fa-4e92-9a39-541cf8983a89-hosts-file\") pod \"node-resolver-x2gfj\" (UID: \"53d63a68-43fa-4e92-9a39-541cf8983a89\") " pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:52:22.843985 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843652    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcbt6\" (UniqueName: \"kubernetes.io/projected/53d63a68-43fa-4e92-9a39-541cf8983a89-kube-api-access-pcbt6\") pod \"node-resolver-x2gfj\" (UID: \"53d63a68-43fa-4e92-9a39-541cf8983a89\") " pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:52:22.843985 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843691    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-system-cni-dir\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.843985 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843714    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.843985 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843733    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-ovn\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.843985 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843755    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-env-overrides\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.843985 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.843776    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4qsd\" (UniqueName: \"kubernetes.io/projected/14434183-f8cd-4f6e-aab4-d8516d44ec74-kube-api-access-t4qsd\") pod \"iptables-alerter-j68lz\" (UID: \"14434183-f8cd-4f6e-aab4-d8516d44ec74\") " pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:52:22.844913 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.844899    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:22.847320 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.847302    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:22.847403 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:22.847360    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4tlxp" podUID="4cced475-7943-49ad-870b-4b7119d62759"
Oct 30 21:52:22.876397 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.876372    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:52:22.877070 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.876947    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:52:22.877174 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.877156    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:52:22.877243 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.877192    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:52:22.877897 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.877875    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:52:22.878097 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.878079    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ww8ls\""
Oct 30 21:52:22.878177 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.878117    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bdqch\""
Oct 30 21:52:22.878237 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.878191    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:52:22.878296 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.878284    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:52:22.878559 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.878543    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:52:22.880546 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.880526    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:52:22.880546 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.880532    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:52:22.880674 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.880539    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:52:22.880674 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.880576    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:52:22.880674 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.880589    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:52:22.880674 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.880637    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q4htg\""
Oct 30 21:52:22.930029 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.929997    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:52:22.930029 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.930020    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:52:22.930251 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.930119    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mgstz\""
Oct 30 21:52:22.930251 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.930120    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:52:22.930251 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.930218    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:52:22.930251 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.930235    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:52:22.930251 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.930250    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:52:22.933755 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.933738    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:52:22.933755 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.933744    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:52:22.934011 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.933892    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:52:22.934011 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.933911    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:52:22.934011 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.933932    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hqgxz\""
Oct 30 21:52:22.934011 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.933939    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:52:22.934011 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.934027    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:52:22.934011 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.934038    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vk8r7\""
Oct 30 21:52:22.934389 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.934375    2539 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:52:22.945110 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945089    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-run-netns\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.945156 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945117    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb4lx\" (UniqueName: \"kubernetes.io/projected/d46f22d3-4503-41af-98d9-c641d343de28-kube-api-access-kb4lx\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.945156 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945136    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-tuned\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.945245 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945164    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbjbq\" (UniqueName: \"kubernetes.io/projected/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-kube-api-access-rbjbq\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:22.945245 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945191    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-host\") pod \"node-ca-pcjh8\" (UID: \"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098\") " pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:52:22.945245 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945213    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-run-netns\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.945245 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945215    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d63a68-43fa-4e92-9a39-541cf8983a89-tmp-dir\") pod \"node-resolver-x2gfj\" (UID: \"53d63a68-43fa-4e92-9a39-541cf8983a89\") " pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:52:22.945394 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945282    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-host\") pod \"node-ca-pcjh8\" (UID: \"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098\") " pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:52:22.945394 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945372    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-registration-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:22.945468 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945424    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-sys-fs\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:22.945523 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945471    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlk69\" (UniqueName: \"kubernetes.io/projected/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-kube-api-access-vlk69\") pod \"node-ca-pcjh8\" (UID: \"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098\") " pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:52:22.945577 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945519    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d63a68-43fa-4e92-9a39-541cf8983a89-tmp-dir\") pod \"node-resolver-x2gfj\" (UID: \"53d63a68-43fa-4e92-9a39-541cf8983a89\") " pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:52:22.945577 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945533    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9cc773b9-a6f5-48bd-9beb-11efeaae63d9-agent-certs\") pod \"konnectivity-agent-2mskp\" (UID: \"9cc773b9-a6f5-48bd-9beb-11efeaae63d9\") " pod="kube-system/konnectivity-agent-2mskp"
Oct 30 21:52:22.945577 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945539    2539 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:52:22.945718 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945573    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9qcl\" (UniqueName: \"kubernetes.io/projected/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-kube-api-access-q9qcl\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:22.945718 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945611    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-host\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.945718 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945640    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-device-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:22.945718 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945669    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-systemd\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.945863 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945714    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-system-cni-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:22.945863 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945715    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-host\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.945863 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945763    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-env-overrides\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.945863 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945774    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-systemd\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.945863 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945781    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4qsd\" (UniqueName: \"kubernetes.io/projected/14434183-f8cd-4f6e-aab4-d8516d44ec74-kube-api-access-t4qsd\") pod \"iptables-alerter-j68lz\" (UID: \"14434183-f8cd-4f6e-aab4-d8516d44ec74\") " pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:52:22.945863 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945813    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysctl-conf\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.945863 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945848    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-cnibin\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:22.946291 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945881    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4cced475-7943-49ad-870b-4b7119d62759-dbus\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:22.946291 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945903    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-sys\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.946291 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.945964    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysctl-conf\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.946291 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946007    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-cni-netd\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.946291 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946029    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-sys\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.946291 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946048    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-systemd-units\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.946291 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946071    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d46f22d3-4503-41af-98d9-c641d343de28-ovn-node-metrics-cert\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.946291 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946074    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-cni-netd\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.946291 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946094    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-conf-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:22.946291 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946103    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-systemd-units\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.946291 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946118    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:22.946291 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946163    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-slash\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.946291 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946198    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-log-socket\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.946291 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946220    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysconfig\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.946291 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946267    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-slash\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.946291 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946274    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysconfig\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.946933 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946304    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89f7427f-84b8-4ee1-a4a6-d901b32851f1-cni-binary-copy\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:22.946933 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946315    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-log-socket\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.946933 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946326    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-socket-dir-parent\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:22.946933 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946369    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9cc773b9-a6f5-48bd-9beb-11efeaae63d9-konnectivity-ca\") pod \"konnectivity-agent-2mskp\" (UID: \"9cc773b9-a6f5-48bd-9beb-11efeaae63d9\") " pod="kube-system/konnectivity-agent-2mskp"
Oct 30 21:52:22.946933 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946402    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-env-overrides\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.946933 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946415    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-serviceca\") pod \"node-ca-pcjh8\" (UID: \"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098\") " pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:52:22.946933 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946441    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-cni-bin\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.946933 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946462    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-kubernetes\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.946933 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946480    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-k8s-cni-cncf-io\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:22.946933 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946497    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-cni-bin\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:22.946933 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946517    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-multus-certs\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:22.946933 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946538    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-etc-kubernetes\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:22.946933 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946565    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-kubernetes\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.946933 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946586    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4cced475-7943-49ad-870b-4b7119d62759-kubelet-config\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:22.946933 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946628    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-node-log\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.946933 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946633    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-cni-bin\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.946933 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946693    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-lib-modules\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.947592 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946721    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-etc-selinux\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:22.947592 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946729    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-node-log\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.947592 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946748    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-cnibin\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.947592 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946774    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-cni-binary-copy\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.947592 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946802    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxs96\" (UniqueName: \"kubernetes.io/projected/6aea3c42-943b-4532-b21d-64ada9922ed2-kube-api-access-kxs96\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.947592 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946821    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-serviceca\") pod \"node-ca-pcjh8\" (UID: \"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098\") " pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:52:22.947592 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946827    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-ovnkube-config\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.947592 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946836    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-cnibin\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.947592 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946849    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-lib-modules\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.947592 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946860    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78e61e36-d7f2-432e-af30-16a093b57e7f-tmp\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.947592 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946888    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5zlk\" (UniqueName: \"kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk\") pod \"network-check-target-b5h24\" (UID: \"60e220ff-82ff-4d25-b53d-6e98dd4da3fd\") " pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:22.947592 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946919    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.947592 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.946947    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-kubelet\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.947592 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947111    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-kubelet\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.947592 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947166    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14434183-f8cd-4f6e-aab4-d8516d44ec74-iptables-alerter-script\") pod \"iptables-alerter-j68lz\" (UID: \"14434183-f8cd-4f6e-aab4-d8516d44ec74\") " pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:52:22.947592 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947353    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14434183-f8cd-4f6e-aab4-d8516d44ec74-host-slash\") pod \"iptables-alerter-j68lz\" (UID: \"14434183-f8cd-4f6e-aab4-d8516d44ec74\") " pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:52:22.947592 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947389    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-run\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.948422 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947399    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-ovnkube-config\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.948422 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947419    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-kubelet\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:22.948422 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947447    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-hostroot\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:22.948422 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947459    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14434183-f8cd-4f6e-aab4-d8516d44ec74-host-slash\") pod \"iptables-alerter-j68lz\" (UID: \"14434183-f8cd-4f6e-aab4-d8516d44ec74\") " pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:52:22.948422 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947485    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnq59\" (UniqueName: \"kubernetes.io/projected/89f7427f-84b8-4ee1-a4a6-d901b32851f1-kube-api-access-cnq59\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:22.948422 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947523    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.948422 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947526    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.948422 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947567    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-var-lib-kubelet\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.948422 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947587    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-run\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.948422 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947596    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53d63a68-43fa-4e92-9a39-541cf8983a89-hosts-file\") pod \"node-resolver-x2gfj\" (UID: \"53d63a68-43fa-4e92-9a39-541cf8983a89\") " pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:52:22.948422 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947629    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.948422 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947644    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53d63a68-43fa-4e92-9a39-541cf8983a89-hosts-file\") pod \"node-resolver-x2gfj\" (UID: \"53d63a68-43fa-4e92-9a39-541cf8983a89\") " pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:52:22.948422 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947661    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcbt6\" (UniqueName: \"kubernetes.io/projected/53d63a68-43fa-4e92-9a39-541cf8983a89-kube-api-access-pcbt6\") pod \"node-resolver-x2gfj\" (UID: \"53d63a68-43fa-4e92-9a39-541cf8983a89\") " pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:52:22.948422 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947710    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-var-lib-kubelet\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.948422 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947783    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-socket-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:22.948422 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947832    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:22.948422 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947901    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-system-cni-dir\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.949188 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947916    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14434183-f8cd-4f6e-aab4-d8516d44ec74-iptables-alerter-script\") pod \"iptables-alerter-j68lz\" (UID: \"14434183-f8cd-4f6e-aab4-d8516d44ec74\") " pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:52:22.949188 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.947996    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.949188 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948017    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-cni-binary-copy\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.949188 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948024    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-ovn\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.949188 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948086    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysctl-d\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.949188 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948103    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-ovn\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.949188 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948112    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz8q9\" (UniqueName: \"kubernetes.io/projected/78e61e36-d7f2-432e-af30-16a093b57e7f-kube-api-access-hz8q9\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.949188 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948169    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-systemd\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.949188 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948183    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysctl-d\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.949188 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948191    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.949188 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948087    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-system-cni-dir\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.949188 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948199    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-var-lib-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.949188 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948236    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-var-lib-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.949188 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948243    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-systemd\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.949188 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948275    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.949188 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948303    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.949188 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948306    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.949686 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948337    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-netns\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:22.949686 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948363    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-cni-multus\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:22.949686 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948435    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-modprobe-d\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.949686 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948469    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-cni-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:22.949686 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948497    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:22.949686 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948516    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-modprobe-d\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.949686 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948524    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-etc-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.949686 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948565    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-run-ovn-kubernetes\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.949686 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948579    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-etc-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.949686 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948596    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-ovnkube-script-lib\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.949686 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948602    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-run-ovn-kubernetes\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.949686 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948628    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-daemon-config\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:22.949686 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.948964    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:22.949686 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.949198    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-ovnkube-script-lib\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.950104 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.949745    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78e61e36-d7f2-432e-af30-16a093b57e7f-tmp\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.950104 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.949786    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-tuned\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:22.950104 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.949834    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d46f22d3-4503-41af-98d9-c641d343de28-ovn-node-metrics-cert\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:22.978431 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.978406    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ppl5h\""
Oct 30 21:52:22.978431 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.978418    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:52:22.978578 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.978438    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:52:22.978578 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.978445    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gg6dx\""
Oct 30 21:52:22.978578 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.978473    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gqgtq\""
Oct 30 21:52:22.978578 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.978488    2539 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:52:22.978578 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:22.978554    2539 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5xkfw"
Oct 30 21:52:23.006747 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.006719    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4qsd\" (UniqueName: \"kubernetes.io/projected/14434183-f8cd-4f6e-aab4-d8516d44ec74-kube-api-access-t4qsd\") pod \"iptables-alerter-j68lz\" (UID: \"14434183-f8cd-4f6e-aab4-d8516d44ec74\") " pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:52:23.006939 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.006915    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb4lx\" (UniqueName: \"kubernetes.io/projected/d46f22d3-4503-41af-98d9-c641d343de28-kube-api-access-kb4lx\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:23.007008 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.006982    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlk69\" (UniqueName: \"kubernetes.io/projected/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-kube-api-access-vlk69\") pod \"node-ca-pcjh8\" (UID: \"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098\") " pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:52:23.007296 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.007278    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxs96\" (UniqueName: \"kubernetes.io/projected/6aea3c42-943b-4532-b21d-64ada9922ed2-kube-api-access-kxs96\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:23.007560 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.007541    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcbt6\" (UniqueName: \"kubernetes.io/projected/53d63a68-43fa-4e92-9a39-541cf8983a89-kube-api-access-pcbt6\") pod \"node-resolver-x2gfj\" (UID: \"53d63a68-43fa-4e92-9a39-541cf8983a89\") " pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:52:23.007904 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.007889    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz8q9\" (UniqueName: \"kubernetes.io/projected/78e61e36-d7f2-432e-af30-16a093b57e7f-kube-api-access-hz8q9\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:23.049601 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.049574    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-netns\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.049601 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.049601    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-cni-multus\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.049764 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.049617    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-cni-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.049764 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.049636    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:23.049764 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.049672    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-daemon-config\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.049764 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.049689    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-cni-multus\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.049764 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.049699    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbjbq\" (UniqueName: \"kubernetes.io/projected/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-kube-api-access-rbjbq\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:23.049764 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.049725    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-registration-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:23.049764 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.049743    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-netns\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.050053 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:23.049781    2539 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:52:23.050053 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.049804    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-cni-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.050053 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.049810    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-sys-fs\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:23.050053 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.049784    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-registration-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:23.050053 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.049865    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-sys-fs\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:23.050053 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:23.049878    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret podName:4cced475-7943-49ad-870b-4b7119d62759 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:23.549835293 +0000 UTC m=+3.191724883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret") pod "global-pull-secret-syncer-4tlxp" (UID: "4cced475-7943-49ad-870b-4b7119d62759") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:52:23.050053 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.049903    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9cc773b9-a6f5-48bd-9beb-11efeaae63d9-agent-certs\") pod \"konnectivity-agent-2mskp\" (UID: \"9cc773b9-a6f5-48bd-9beb-11efeaae63d9\") " pod="kube-system/konnectivity-agent-2mskp"
Oct 30 21:52:23.050053 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.049928    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9qcl\" (UniqueName: \"kubernetes.io/projected/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-kube-api-access-q9qcl\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:23.050053 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.049960    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-device-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:23.050053 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.049998    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-system-cni-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.050053 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050019    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-cnibin\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.050053 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050040    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4cced475-7943-49ad-870b-4b7119d62759-dbus\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050070    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-conf-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050082    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-system-cni-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050092    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-device-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050124    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050140    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-conf-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050147    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4cced475-7943-49ad-870b-4b7119d62759-dbus\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050154    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89f7427f-84b8-4ee1-a4a6-d901b32851f1-cni-binary-copy\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050181    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-socket-dir-parent\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050189    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050218    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9cc773b9-a6f5-48bd-9beb-11efeaae63d9-konnectivity-ca\") pod \"konnectivity-agent-2mskp\" (UID: \"9cc773b9-a6f5-48bd-9beb-11efeaae63d9\") " pod="kube-system/konnectivity-agent-2mskp"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050235    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-socket-dir-parent\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050253    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-daemon-config\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050344    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-cnibin\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050393    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-k8s-cni-cncf-io\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050421    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-cni-bin\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050439    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-multus-certs\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050458    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-etc-kubernetes\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.050621 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050466    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-k8s-cni-cncf-io\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050476    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4cced475-7943-49ad-870b-4b7119d62759-kubelet-config\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050516    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-cni-bin\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050528    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-etc-kubernetes\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050537    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-etc-selinux\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050555    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-multus-certs\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050626    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5zlk\" (UniqueName: \"kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk\") pod \"network-check-target-b5h24\" (UID: \"60e220ff-82ff-4d25-b53d-6e98dd4da3fd\") " pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050646    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-etc-selinux\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050649    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-kubelet\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050668    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-kubelet\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050685    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-hostroot\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050691    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89f7427f-84b8-4ee1-a4a6-d901b32851f1-cni-binary-copy\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050714    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnq59\" (UniqueName: \"kubernetes.io/projected/89f7427f-84b8-4ee1-a4a6-d901b32851f1-kube-api-access-cnq59\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050761    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-hostroot\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050792    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9cc773b9-a6f5-48bd-9beb-11efeaae63d9-konnectivity-ca\") pod \"konnectivity-agent-2mskp\" (UID: \"9cc773b9-a6f5-48bd-9beb-11efeaae63d9\") " pod="kube-system/konnectivity-agent-2mskp"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050863    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4cced475-7943-49ad-870b-4b7119d62759-kubelet-config\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050900    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-socket-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.050920    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:23.051338 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:23.051028    2539 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:52:23.051847 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.051048    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-socket-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:23.051847 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:23.051074    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs podName:dbf75704-af7b-44f6-bfcb-b2b2e943f5e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:23.551062173 +0000 UTC m=+3.192951761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs") pod "network-metrics-daemon-kprst" (UID: "dbf75704-af7b-44f6-bfcb-b2b2e943f5e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:52:23.052597 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.052581    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9cc773b9-a6f5-48bd-9beb-11efeaae63d9-agent-certs\") pod \"konnectivity-agent-2mskp\" (UID: \"9cc773b9-a6f5-48bd-9beb-11efeaae63d9\") " pod="kube-system/konnectivity-agent-2mskp"
Oct 30 21:52:23.113019 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:23.112993    2539 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:52:23.113019 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:23.113019    2539 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:52:23.113202 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:23.113032    2539 projected.go:196] Error preparing data for projected volume kube-api-access-l5zlk for pod openshift-network-diagnostics/network-check-target-b5h24: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:52:23.113202 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:23.113100    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk podName:60e220ff-82ff-4d25-b53d-6e98dd4da3fd nodeName:}" failed. No retries permitted until 2026-10-30 21:52:23.613084509 +0000 UTC m=+3.254974094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l5zlk" (UniqueName: "kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk") pod "network-check-target-b5h24" (UID: "60e220ff-82ff-4d25-b53d-6e98dd4da3fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:52:23.114699 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.114679    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbjbq\" (UniqueName: \"kubernetes.io/projected/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-kube-api-access-rbjbq\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:23.114763 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.114680    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9qcl\" (UniqueName: \"kubernetes.io/projected/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-kube-api-access-q9qcl\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:23.115167 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.115146    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnq59\" (UniqueName: \"kubernetes.io/projected/89f7427f-84b8-4ee1-a4a6-d901b32851f1-kube-api-access-cnq59\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.144069 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.143994    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:52:23.149448 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.149428    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:52:23.156093 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:23.156068    2539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60ffcff_4c40_4ae9_b5bd_2e23bf3bd098.slice/crio-a97e2000b390e946e0c1ce5f08cda7bb01b9b865a0cd679501f9abfe06639228 WatchSource:0}: Error finding container a97e2000b390e946e0c1ce5f08cda7bb01b9b865a0cd679501f9abfe06639228: Status 404 returned error can't find the container with id a97e2000b390e946e0c1ce5f08cda7bb01b9b865a0cd679501f9abfe06639228
Oct 30 21:52:23.159474 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.159404    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:52:23.159548 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.159506    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:52:23.168897 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.168878    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:52:23.170536 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:23.170506    2539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14434183_f8cd_4f6e_aab4_d8516d44ec74.slice/crio-972bcb06f95cbf6872be72074ef4d8708153b5e6409ccfdd4a945ae8ccd21f88 WatchSource:0}: Error finding container 972bcb06f95cbf6872be72074ef4d8708153b5e6409ccfdd4a945ae8ccd21f88: Status 404 returned error can't find the container with id 972bcb06f95cbf6872be72074ef4d8708153b5e6409ccfdd4a945ae8ccd21f88
Oct 30 21:52:23.174791 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:23.174767    2539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e61e36_d7f2_432e_af30_16a093b57e7f.slice/crio-5b0879ff4e46dc5e2ee522d211ebfc490e6da6d9f33a9f369c8a1ed179464caf WatchSource:0}: Error finding container 5b0879ff4e46dc5e2ee522d211ebfc490e6da6d9f33a9f369c8a1ed179464caf: Status 404 returned error can't find the container with id 5b0879ff4e46dc5e2ee522d211ebfc490e6da6d9f33a9f369c8a1ed179464caf
Oct 30 21:52:23.175546 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.175528    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:23.179470 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:23.179443    2539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d63a68_43fa_4e92_9a39_541cf8983a89.slice/crio-6c04ba2f75948bec668a42b2992868bcf84922d629761f0bbe67ca23928332c4 WatchSource:0}: Error finding container 6c04ba2f75948bec668a42b2992868bcf84922d629761f0bbe67ca23928332c4: Status 404 returned error can't find the container with id 6c04ba2f75948bec668a42b2992868bcf84922d629761f0bbe67ca23928332c4
Oct 30 21:52:23.182695 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.182669    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2mskp"
Oct 30 21:52:23.185086 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.185045    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lh5dm"
Oct 30 21:52:23.185193 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:23.185173    2539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46f22d3_4503_41af_98d9_c641d343de28.slice/crio-5a48f337c08359807dacc297270f3d58bd1ca9b2989bc691756e9434805a5416 WatchSource:0}: Error finding container 5a48f337c08359807dacc297270f3d58bd1ca9b2989bc691756e9434805a5416: Status 404 returned error can't find the container with id 5a48f337c08359807dacc297270f3d58bd1ca9b2989bc691756e9434805a5416
Oct 30 21:52:23.187257 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.187229    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:52:23.194601 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:23.194581    2539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc773b9_a6f5_48bd_9beb_11efeaae63d9.slice/crio-a2d7957db3c761ac03f31b0a0e04b1010cc70c3ddb947ec72fee2ded6e981b84 WatchSource:0}: Error finding container a2d7957db3c761ac03f31b0a0e04b1010cc70c3ddb947ec72fee2ded6e981b84: Status 404 returned error can't find the container with id a2d7957db3c761ac03f31b0a0e04b1010cc70c3ddb947ec72fee2ded6e981b84
Oct 30 21:52:23.196076 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:23.196051    2539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f7427f_84b8_4ee1_a4a6_d901b32851f1.slice/crio-95ba7a02912e118a2301ba12ffb28f2abc918aa5ec297848b8a957cff6235870 WatchSource:0}: Error finding container 95ba7a02912e118a2301ba12ffb28f2abc918aa5ec297848b8a957cff6235870: Status 404 returned error can't find the container with id 95ba7a02912e118a2301ba12ffb28f2abc918aa5ec297848b8a957cff6235870
Oct 30 21:52:23.199227 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:23.199209    2539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba18036_6a0c_4aa6_a6b3_b9ab35a57004.slice/crio-e16268348e4444fd54c6ce3d29ad7ebd08975886f2cfd6fa7097122dd8712a95 WatchSource:0}: Error finding container e16268348e4444fd54c6ce3d29ad7ebd08975886f2cfd6fa7097122dd8712a95: Status 404 returned error can't find the container with id e16268348e4444fd54c6ce3d29ad7ebd08975886f2cfd6fa7097122dd8712a95
Oct 30 21:52:23.555469 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.555428    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:23.555658 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.555485    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:23.555658 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:23.555613    2539 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:52:23.555658 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:23.555614    2539 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:52:23.555773 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:23.555676    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret podName:4cced475-7943-49ad-870b-4b7119d62759 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:24.555656149 +0000 UTC m=+4.197545756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret") pod "global-pull-secret-syncer-4tlxp" (UID: "4cced475-7943-49ad-870b-4b7119d62759") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:52:23.555773 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:23.555692    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs podName:dbf75704-af7b-44f6-bfcb-b2b2e943f5e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:24.555684623 +0000 UTC m=+4.197574207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs") pod "network-metrics-daemon-kprst" (UID: "dbf75704-af7b-44f6-bfcb-b2b2e943f5e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:52:23.655853 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.655815    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5zlk\" (UniqueName: \"kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk\") pod \"network-check-target-b5h24\" (UID: \"60e220ff-82ff-4d25-b53d-6e98dd4da3fd\") " pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:23.656122 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:23.656028    2539 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:52:23.656122 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:23.656052    2539 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:52:23.656122 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:23.656066    2539 projected.go:196] Error preparing data for projected volume kube-api-access-l5zlk for pod openshift-network-diagnostics/network-check-target-b5h24: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:52:23.656335 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:23.656136    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk podName:60e220ff-82ff-4d25-b53d-6e98dd4da3fd nodeName:}" failed. No retries permitted until 2026-10-30 21:52:24.656116176 +0000 UTC m=+4.298005777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5zlk" (UniqueName: "kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk") pod "network-check-target-b5h24" (UID: "60e220ff-82ff-4d25-b53d-6e98dd4da3fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:52:23.817867 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.817785    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28mbh" event={"ID":"6aea3c42-943b-4532-b21d-64ada9922ed2","Type":"ContainerStarted","Data":"319b5d7d5b2e047e272d6e4639deac2c71c67f1c68a6c672d52f330b0b7bc780"}
Oct 30 21:52:23.819840 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.819339    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pcjh8" event={"ID":"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098","Type":"ContainerStarted","Data":"a97e2000b390e946e0c1ce5f08cda7bb01b9b865a0cd679501f9abfe06639228"}
Oct 30 21:52:23.820685 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.820656    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb" event={"ID":"fba18036-6a0c-4aa6-a6b3-b9ab35a57004","Type":"ContainerStarted","Data":"e16268348e4444fd54c6ce3d29ad7ebd08975886f2cfd6fa7097122dd8712a95"}
Oct 30 21:52:23.822256 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.822220    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lh5dm" event={"ID":"89f7427f-84b8-4ee1-a4a6-d901b32851f1","Type":"ContainerStarted","Data":"95ba7a02912e118a2301ba12ffb28f2abc918aa5ec297848b8a957cff6235870"}
Oct 30 21:52:23.823511 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.823477    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2mskp" event={"ID":"9cc773b9-a6f5-48bd-9beb-11efeaae63d9","Type":"ContainerStarted","Data":"a2d7957db3c761ac03f31b0a0e04b1010cc70c3ddb947ec72fee2ded6e981b84"}
Oct 30 21:52:23.824670 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.824651    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x2gfj" event={"ID":"53d63a68-43fa-4e92-9a39-541cf8983a89","Type":"ContainerStarted","Data":"6c04ba2f75948bec668a42b2992868bcf84922d629761f0bbe67ca23928332c4"}
Oct 30 21:52:23.825818 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.825795    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7hbsq" event={"ID":"78e61e36-d7f2-432e-af30-16a093b57e7f","Type":"ContainerStarted","Data":"5b0879ff4e46dc5e2ee522d211ebfc490e6da6d9f33a9f369c8a1ed179464caf"}
Oct 30 21:52:23.827076 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.827055    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j68lz" event={"ID":"14434183-f8cd-4f6e-aab4-d8516d44ec74","Type":"ContainerStarted","Data":"972bcb06f95cbf6872be72074ef4d8708153b5e6409ccfdd4a945ae8ccd21f88"}
Oct 30 21:52:23.828234 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.828214    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5" event={"ID":"d46f22d3-4503-41af-98d9-c641d343de28","Type":"ContainerStarted","Data":"5a48f337c08359807dacc297270f3d58bd1ca9b2989bc691756e9434805a5416"}
Oct 30 21:52:23.982755 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.982655    2539 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:47:23 +0000 UTC" deadline="2028-08-02 00:34:50.001756624 +0000 UTC"
Oct 30 21:52:23.982755 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:23.982752    2539 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15386h42m26.019009554s"
Oct 30 21:52:24.564437 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:24.564404    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:24.564614 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:24.564460    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:24.564679 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:24.564625    2539 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:52:24.564749 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:24.564687    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret podName:4cced475-7943-49ad-870b-4b7119d62759 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:26.564668373 +0000 UTC m=+6.206557975 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret") pod "global-pull-secret-syncer-4tlxp" (UID: "4cced475-7943-49ad-870b-4b7119d62759") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:52:24.564828 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:24.564758    2539 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:52:24.564828 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:24.564793    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs podName:dbf75704-af7b-44f6-bfcb-b2b2e943f5e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:26.564780029 +0000 UTC m=+6.206669629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs") pod "network-metrics-daemon-kprst" (UID: "dbf75704-af7b-44f6-bfcb-b2b2e943f5e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:52:24.666308 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:24.665329    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5zlk\" (UniqueName: \"kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk\") pod \"network-check-target-b5h24\" (UID: \"60e220ff-82ff-4d25-b53d-6e98dd4da3fd\") " pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:24.666308 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:24.665509    2539 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:52:24.666308 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:24.665533    2539 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:52:24.666308 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:24.665545    2539 projected.go:196] Error preparing data for projected volume kube-api-access-l5zlk for pod openshift-network-diagnostics/network-check-target-b5h24: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:52:24.666308 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:24.665614    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk podName:60e220ff-82ff-4d25-b53d-6e98dd4da3fd nodeName:}" failed. No retries permitted until 2026-10-30 21:52:26.665584301 +0000 UTC m=+6.307473907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5zlk" (UniqueName: "kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk") pod "network-check-target-b5h24" (UID: "60e220ff-82ff-4d25-b53d-6e98dd4da3fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:52:24.811093 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:24.807086    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:24.811093 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:24.807214    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5h24" podUID="60e220ff-82ff-4d25-b53d-6e98dd4da3fd"
Oct 30 21:52:24.811093 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:24.807302    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:24.811093 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:24.807399    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kprst" podUID="dbf75704-af7b-44f6-bfcb-b2b2e943f5e6"
Oct 30 21:52:24.811093 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:24.807468    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:24.811093 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:24.807549    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4tlxp" podUID="4cced475-7943-49ad-870b-4b7119d62759"
Oct 30 21:52:24.985618 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:24.983617    2539 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:47:23 +0000 UTC" deadline="2028-06-24 11:15:16.610944388 +0000 UTC"
Oct 30 21:52:24.985618 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:24.983653    2539 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14461h22m51.62729863s"
Oct 30 21:52:25.853569 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:25.853297    2539 generic.go:356] "Generic (PLEG): container finished" podID="0a78e001a5670be0097c09eab27b6eea" containerID="b3ff85605e6aca3f01e12ca75d477dcc0aa99ab199f9cbf5d6f482be3d5bbccd" exitCode=0
Oct 30 21:52:25.854355 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:25.854015    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal" event={"ID":"0a78e001a5670be0097c09eab27b6eea","Type":"ContainerDied","Data":"b3ff85605e6aca3f01e12ca75d477dcc0aa99ab199f9cbf5d6f482be3d5bbccd"}
Oct 30 21:52:25.872145 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:25.871758    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal" event={"ID":"a49dc1ff1f4db21ad91c65b8f36133e5","Type":"ContainerStarted","Data":"ea5d13d1a468b967b7b150db725bb8829122c9304dddf7d920a58a88d616f435"}
Oct 30 21:52:26.079628 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:26.079559    2539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal" podStartSLOduration=4.079540303 podStartE2EDuration="4.079540303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:26.079020302 +0000 UTC m=+5.720909910" watchObservedRunningTime="2026-10-30 21:52:26.079540303 +0000 UTC m=+5.721429909"
Oct 30 21:52:26.580465 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:26.580008    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:26.580465 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:26.580058    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:26.580465 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:26.580208    2539 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:52:26.580465 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:26.580255    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret podName:4cced475-7943-49ad-870b-4b7119d62759 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:30.580243453 +0000 UTC m=+10.222133048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret") pod "global-pull-secret-syncer-4tlxp" (UID: "4cced475-7943-49ad-870b-4b7119d62759") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:52:26.580465 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:26.580318    2539 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:52:26.580465 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:26.580353    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs podName:dbf75704-af7b-44f6-bfcb-b2b2e943f5e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:30.580341972 +0000 UTC m=+10.222231568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs") pod "network-metrics-daemon-kprst" (UID: "dbf75704-af7b-44f6-bfcb-b2b2e943f5e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:52:26.680693 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:26.680660    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5zlk\" (UniqueName: \"kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk\") pod \"network-check-target-b5h24\" (UID: \"60e220ff-82ff-4d25-b53d-6e98dd4da3fd\") " pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:26.680873 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:26.680851    2539 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:52:26.680940 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:26.680880    2539 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:52:26.680940 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:26.680893    2539 projected.go:196] Error preparing data for projected volume kube-api-access-l5zlk for pod openshift-network-diagnostics/network-check-target-b5h24: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:52:26.681059 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:26.680968    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk podName:60e220ff-82ff-4d25-b53d-6e98dd4da3fd nodeName:}" failed. No retries permitted until 2026-10-30 21:52:30.680933643 +0000 UTC m=+10.322823240 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5zlk" (UniqueName: "kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk") pod "network-check-target-b5h24" (UID: "60e220ff-82ff-4d25-b53d-6e98dd4da3fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:52:26.809238 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:26.807055    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:26.809238 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:26.807190    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4tlxp" podUID="4cced475-7943-49ad-870b-4b7119d62759"
Oct 30 21:52:26.809238 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:26.807256    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:26.809238 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:26.807319    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5h24" podUID="60e220ff-82ff-4d25-b53d-6e98dd4da3fd"
Oct 30 21:52:26.809238 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:26.807453    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:26.809238 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:26.807542    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kprst" podUID="dbf75704-af7b-44f6-bfcb-b2b2e943f5e6"
Oct 30 21:52:26.876605 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:26.876571    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal" event={"ID":"0a78e001a5670be0097c09eab27b6eea","Type":"ContainerStarted","Data":"8a2ba6703ff5bc2739f8d5b2f08b18b481be78e494a0d181211112f4b17728f0"}
Oct 30 21:52:26.879123 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:26.879094    2539 generic.go:356] "Generic (PLEG): container finished" podID="6aea3c42-943b-4532-b21d-64ada9922ed2" containerID="95bec18c53211db4775e411555b05cbfe9635dc700a4aded3d0f25d7ca88c4d0" exitCode=0
Oct 30 21:52:26.880054 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:26.880031    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28mbh" event={"ID":"6aea3c42-943b-4532-b21d-64ada9922ed2","Type":"ContainerDied","Data":"95bec18c53211db4775e411555b05cbfe9635dc700a4aded3d0f25d7ca88c4d0"}
Oct 30 21:52:26.989373 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:26.989275    2539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal" podStartSLOduration=4.989258575 podStartE2EDuration="4.989258575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:26.988743193 +0000 UTC m=+6.630632800" watchObservedRunningTime="2026-10-30 21:52:26.989258575 +0000 UTC m=+6.631148181"
Oct 30 21:52:28.807842 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:28.807757    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:28.807842 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:28.807781    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:28.807842 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:28.807811    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:28.808468 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:28.807903    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5h24" podUID="60e220ff-82ff-4d25-b53d-6e98dd4da3fd"
Oct 30 21:52:28.808468 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:28.808004    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kprst" podUID="dbf75704-af7b-44f6-bfcb-b2b2e943f5e6"
Oct 30 21:52:28.808468 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:28.808095    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4tlxp" podUID="4cced475-7943-49ad-870b-4b7119d62759"
Oct 30 21:52:28.894047 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:28.893094    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pcjh8" event={"ID":"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098","Type":"ContainerStarted","Data":"6b554acb0eba5289fc7b2019664a4913328ac1cd75eb206809a648085db4f471"}
Oct 30 21:52:28.899779 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:28.899370    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2mskp" event={"ID":"9cc773b9-a6f5-48bd-9beb-11efeaae63d9","Type":"ContainerStarted","Data":"f462732800c16127042818aaa4f0de95fa94551b1dc40cf3e7dcbf5ee73b8081"}
Oct 30 21:52:28.934386 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:28.934356    2539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2mskp"
Oct 30 21:52:28.935049 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:28.935027    2539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2mskp"
Oct 30 21:52:29.127231 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:29.126285    2539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pcjh8" podStartSLOduration=2.793998944 podStartE2EDuration="8.126266245s" totalImagesPullingTime="5.332267301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:29.012277602 +0000 UTC m=+8.654167212" watchObservedRunningTime="2026-10-30 21:52:29.126266245 +0000 UTC m=+8.768155858"
Oct 30 21:52:29.908667 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:29.908630    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb" event={"ID":"fba18036-6a0c-4aa6-a6b3-b9ab35a57004","Type":"ContainerStarted","Data":"3b1c2beef44e1b7c96e9090cc7a4da05376ad57b5c9aa8288af3104546bc87a2"}
Oct 30 21:52:30.612812 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:30.612766    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:30.612989 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:30.612831    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:30.613068 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:30.613026    2539 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:52:30.613123 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:30.613097    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret podName:4cced475-7943-49ad-870b-4b7119d62759 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:38.61307968 +0000 UTC m=+18.254969278 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret") pod "global-pull-secret-syncer-4tlxp" (UID: "4cced475-7943-49ad-870b-4b7119d62759") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:52:30.613552 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:30.613516    2539 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:52:30.613655 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:30.613576    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs podName:dbf75704-af7b-44f6-bfcb-b2b2e943f5e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:38.613561792 +0000 UTC m=+18.255451394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs") pod "network-metrics-daemon-kprst" (UID: "dbf75704-af7b-44f6-bfcb-b2b2e943f5e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:52:30.713279 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:30.713238    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5zlk\" (UniqueName: \"kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk\") pod \"network-check-target-b5h24\" (UID: \"60e220ff-82ff-4d25-b53d-6e98dd4da3fd\") " pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:30.713494 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:30.713476    2539 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:52:30.713534 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:30.713508    2539 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:52:30.713534 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:30.713522    2539 projected.go:196] Error preparing data for projected volume kube-api-access-l5zlk for pod openshift-network-diagnostics/network-check-target-b5h24: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:52:30.713608 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:30.713595    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk podName:60e220ff-82ff-4d25-b53d-6e98dd4da3fd nodeName:}" failed. No retries permitted until 2026-10-30 21:52:38.71357684 +0000 UTC m=+18.355466438 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5zlk" (UniqueName: "kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk") pod "network-check-target-b5h24" (UID: "60e220ff-82ff-4d25-b53d-6e98dd4da3fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:52:30.817983 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:30.813767    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:30.817983 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:30.813903    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5h24" podUID="60e220ff-82ff-4d25-b53d-6e98dd4da3fd"
Oct 30 21:52:30.817983 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:30.814296    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:30.817983 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:30.814393    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kprst" podUID="dbf75704-af7b-44f6-bfcb-b2b2e943f5e6"
Oct 30 21:52:30.817983 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:30.814444    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:30.817983 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:30.814510    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4tlxp" podUID="4cced475-7943-49ad-870b-4b7119d62759"
Oct 30 21:52:30.910888 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:30.910859    2539 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:52:31.915362 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:31.915332    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lh5dm" event={"ID":"89f7427f-84b8-4ee1-a4a6-d901b32851f1","Type":"ContainerStarted","Data":"3dd9990f4afe77541337f03658dd91272fb8db5be753b794cc756f443c487414"}
Oct 30 21:52:31.918823 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:31.918790    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x2gfj" event={"ID":"53d63a68-43fa-4e92-9a39-541cf8983a89","Type":"ContainerStarted","Data":"9b93dac26f62385e56b4d5f8935f6366ae93258cb03ca9ca0840c9e2600e8f04"}
Oct 30 21:52:31.935585 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:31.935564    2539 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:52:32.033372 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:32.033109    2539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2mskp" podStartSLOduration=6.331011671 podStartE2EDuration="11.033089741s" totalImagesPullingTime="4.70207807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:29.127170244 +0000 UTC m=+8.769059852" watchObservedRunningTime="2026-10-30 21:52:32.033089741 +0000 UTC m=+11.674979347"
Oct 30 21:52:32.150299 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:32.150234    2539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-lh5dm" podStartSLOduration=2.645358784 podStartE2EDuration="11.150212887s" totalImagesPullingTime="8.504854103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:32.034161046 +0000 UTC m=+11.676050651" watchObservedRunningTime="2026-10-30 21:52:32.150212887 +0000 UTC m=+11.792102493"
Oct 30 21:52:32.808103 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:32.808059    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:32.808103 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:32.808085    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:32.808342 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:32.808196    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4tlxp" podUID="4cced475-7943-49ad-870b-4b7119d62759"
Oct 30 21:52:32.808342 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:32.808268    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:32.808445 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:32.808347    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5h24" podUID="60e220ff-82ff-4d25-b53d-6e98dd4da3fd"
Oct 30 21:52:32.808445 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:32.808421    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kprst" podUID="dbf75704-af7b-44f6-bfcb-b2b2e943f5e6"
Oct 30 21:52:32.819212 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:32.819100    2539 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:52:31.93558512Z","UUID":"2152c8e3-cb5b-4cdc-b50f-0c7c09b0e7c8","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:52:32.821085 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:32.821064    2539 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:52:32.821085 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:32.821089    2539 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:52:32.924007 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:32.923943    2539 generic.go:356] "Generic (PLEG): container finished" podID="6aea3c42-943b-4532-b21d-64ada9922ed2" containerID="4185b187c2416617b38e6600f8415d1c11ae696faf527f288ed32f74aac8bbf4" exitCode=0
Oct 30 21:52:32.924400 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:32.924060    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28mbh" event={"ID":"6aea3c42-943b-4532-b21d-64ada9922ed2","Type":"ContainerDied","Data":"4185b187c2416617b38e6600f8415d1c11ae696faf527f288ed32f74aac8bbf4"}
Oct 30 21:52:32.928887 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:32.928863    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb" event={"ID":"fba18036-6a0c-4aa6-a6b3-b9ab35a57004","Type":"ContainerStarted","Data":"6be9a92a19481c8f7c248afaf51ad14f8e5da674eaf67f38436d65ced9e8cdac"}
Oct 30 21:52:33.061431 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:33.060905    2539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-x2gfj" podStartSLOduration=3.685962794 podStartE2EDuration="12.060886408s" totalImagesPullingTime="8.374923614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:32.146119547 +0000 UTC m=+11.788009153" watchObservedRunningTime="2026-10-30 21:52:33.060886408 +0000 UTC m=+12.702776017"
Oct 30 21:52:33.933321 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:33.933285    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb" event={"ID":"fba18036-6a0c-4aa6-a6b3-b9ab35a57004","Type":"ContainerStarted","Data":"85355fc16c2b5bd75f55520a82c4bb0aea425a3bb11b34f4e5a43c64b1594197"}
Oct 30 21:52:33.934720 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:33.934685    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j68lz" event={"ID":"14434183-f8cd-4f6e-aab4-d8516d44ec74","Type":"ContainerStarted","Data":"9ba3d2b202ce26648bc8a59c63768e7e13c1fa72fd7ccf76ec8b5412433c3315"}
Oct 30 21:52:34.098595 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:34.098546    2539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb" podStartSLOduration=3.382257869 podStartE2EDuration="13.098526174s" totalImagesPullingTime="9.716268305s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:52:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:34.048929655 +0000 UTC m=+13.690819273" watchObservedRunningTime="2026-10-30 21:52:34.098526174 +0000 UTC m=+13.740415781"
Oct 30 21:52:34.166683 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:34.166025    2539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j68lz" podStartSLOduration=4.394302901 podStartE2EDuration="13.166005334s" totalImagesPullingTime="8.771702433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:34.165472851 +0000 UTC m=+13.807362456" watchObservedRunningTime="2026-10-30 21:52:34.166005334 +0000 UTC m=+13.807894942"
Oct 30 21:52:34.807656 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:34.807584    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:34.807656 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:34.807618    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:34.807868 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:34.807589    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:34.807868 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:34.807731    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kprst" podUID="dbf75704-af7b-44f6-bfcb-b2b2e943f5e6"
Oct 30 21:52:34.807868 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:34.807801    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4tlxp" podUID="4cced475-7943-49ad-870b-4b7119d62759"
Oct 30 21:52:34.808020 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:34.807880    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5h24" podUID="60e220ff-82ff-4d25-b53d-6e98dd4da3fd"
Oct 30 21:52:34.938572 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:34.938535    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7hbsq" event={"ID":"78e61e36-d7f2-432e-af30-16a093b57e7f","Type":"ContainerStarted","Data":"54ff014d9af183da6c373ac11ba09b2d8cde973d2bcd30f41c991d08036f6501"}
Oct 30 21:52:34.941283 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:34.941256    2539 generic.go:356] "Generic (PLEG): container finished" podID="6aea3c42-943b-4532-b21d-64ada9922ed2" containerID="acc518ef55e5bb300989ec7be340d37b706f99f6382ab39c01efb7116cdf308d" exitCode=0
Oct 30 21:52:34.941410 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:34.941345    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28mbh" event={"ID":"6aea3c42-943b-4532-b21d-64ada9922ed2","Type":"ContainerDied","Data":"acc518ef55e5bb300989ec7be340d37b706f99f6382ab39c01efb7116cdf308d"}
Oct 30 21:52:35.055943 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:35.055896    2539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7hbsq" podStartSLOduration=3.156252207 podStartE2EDuration="14.055882507s" totalImagesPullingTime="10.8996303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:35.055340927 +0000 UTC m=+14.697230533" watchObservedRunningTime="2026-10-30 21:52:35.055882507 +0000 UTC m=+14.697772112"
Oct 30 21:52:36.807838 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:36.807801    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:36.807838 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:36.807836    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:36.808434 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:36.807941    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:36.808434 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:36.807973    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kprst" podUID="dbf75704-af7b-44f6-bfcb-b2b2e943f5e6"
Oct 30 21:52:36.808434 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:36.808063    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4tlxp" podUID="4cced475-7943-49ad-870b-4b7119d62759"
Oct 30 21:52:36.808434 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:36.808133    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5h24" podUID="60e220ff-82ff-4d25-b53d-6e98dd4da3fd"
Oct 30 21:52:36.949300 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:36.949046    2539 generic.go:356] "Generic (PLEG): container finished" podID="6aea3c42-943b-4532-b21d-64ada9922ed2" containerID="4a772e2b9c1ff51b251ca31fd951313b698044e084d73612cdf50cd959a428a7" exitCode=0
Oct 30 21:52:36.949300 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:36.949152    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28mbh" event={"ID":"6aea3c42-943b-4532-b21d-64ada9922ed2","Type":"ContainerDied","Data":"4a772e2b9c1ff51b251ca31fd951313b698044e084d73612cdf50cd959a428a7"}
Oct 30 21:52:38.670462 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:38.670425    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:38.670975 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:38.670469    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:38.670975 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:38.670584    2539 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:52:38.670975 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:38.670633    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret podName:4cced475-7943-49ad-870b-4b7119d62759 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:54.670617906 +0000 UTC m=+34.312507490 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret") pod "global-pull-secret-syncer-4tlxp" (UID: "4cced475-7943-49ad-870b-4b7119d62759") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:52:38.670975 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:38.670696    2539 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:52:38.670975 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:38.670724    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs podName:dbf75704-af7b-44f6-bfcb-b2b2e943f5e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:54.670714273 +0000 UTC m=+34.312603856 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs") pod "network-metrics-daemon-kprst" (UID: "dbf75704-af7b-44f6-bfcb-b2b2e943f5e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:52:38.715211 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:38.715185    2539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2mskp"
Oct 30 21:52:38.715341 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:38.715327    2539 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:52:38.716120 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:38.716098    2539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2mskp"
Oct 30 21:52:38.771198 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:38.771165    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5zlk\" (UniqueName: \"kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk\") pod \"network-check-target-b5h24\" (UID: \"60e220ff-82ff-4d25-b53d-6e98dd4da3fd\") " pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:38.771388 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:38.771369    2539 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:52:38.771461 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:38.771392    2539 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:52:38.771461 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:38.771401    2539 projected.go:196] Error preparing data for projected volume kube-api-access-l5zlk for pod openshift-network-diagnostics/network-check-target-b5h24: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:52:38.771562 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:38.771462    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk podName:60e220ff-82ff-4d25-b53d-6e98dd4da3fd nodeName:}" failed. No retries permitted until 2026-10-30 21:52:54.771444654 +0000 UTC m=+34.413334255 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5zlk" (UniqueName: "kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk") pod "network-check-target-b5h24" (UID: "60e220ff-82ff-4d25-b53d-6e98dd4da3fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:52:38.807375 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:38.807339    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:38.807375 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:38.807387    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:38.807577 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:38.807350    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:38.807577 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:38.807473    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kprst" podUID="dbf75704-af7b-44f6-bfcb-b2b2e943f5e6"
Oct 30 21:52:38.807577 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:38.807531    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5h24" podUID="60e220ff-82ff-4d25-b53d-6e98dd4da3fd"
Oct 30 21:52:38.807667 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:38.807627    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4tlxp" podUID="4cced475-7943-49ad-870b-4b7119d62759"
Oct 30 21:52:38.957015 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:38.956625    2539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 21:52:38.957379 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:38.957360    2539 generic.go:356] "Generic (PLEG): container finished" podID="d46f22d3-4503-41af-98d9-c641d343de28" containerID="d663386d48bee40654cd125a6da85b38dd8605fb434953a9ab8a4a084ffafd5c" exitCode=1
Oct 30 21:52:38.957477 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:38.957437    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5" event={"ID":"d46f22d3-4503-41af-98d9-c641d343de28","Type":"ContainerStarted","Data":"f43f5fd83e3933c78963fcbb7d2f234394a8dc07531949fca6baed800bea3a94"}
Oct 30 21:52:38.957532 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:38.957506    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5" event={"ID":"d46f22d3-4503-41af-98d9-c641d343de28","Type":"ContainerStarted","Data":"391bd2ac968bfec5a72a46071954826303dec4ad3548cad6814ea0351665dbcd"}
Oct 30 21:52:38.957594 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:38.957580    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5" event={"ID":"d46f22d3-4503-41af-98d9-c641d343de28","Type":"ContainerStarted","Data":"c5f879175b6f7cbb0c704f8711d7c7d363589cc97d8295999b56db3fa3531789"}
Oct 30 21:52:38.957648 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:38.957631    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5" event={"ID":"d46f22d3-4503-41af-98d9-c641d343de28","Type":"ContainerStarted","Data":"a5ef761f4aef192f3267dd79532fa92f03c8afad3f6e79423acaea0fa833789e"}
Oct 30 21:52:38.957685 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:38.957678    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5" event={"ID":"d46f22d3-4503-41af-98d9-c641d343de28","Type":"ContainerDied","Data":"d663386d48bee40654cd125a6da85b38dd8605fb434953a9ab8a4a084ffafd5c"}
Oct 30 21:52:38.957747 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:38.957727    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5" event={"ID":"d46f22d3-4503-41af-98d9-c641d343de28","Type":"ContainerStarted","Data":"5c419a259ca2e3c480f36a85ba2128d07bdeaa925ada7855a482607ca814f5ae"}
Oct 30 21:52:39.965250 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:39.965213    2539 generic.go:356] "Generic (PLEG): container finished" podID="6aea3c42-943b-4532-b21d-64ada9922ed2" containerID="0aedaaa6edce2f98ba97c6c453e9f1f9c77c6f6e6eabdaca3296d5acc2f2bca2" exitCode=0
Oct 30 21:52:39.965641 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:39.965277    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28mbh" event={"ID":"6aea3c42-943b-4532-b21d-64ada9922ed2","Type":"ContainerDied","Data":"0aedaaa6edce2f98ba97c6c453e9f1f9c77c6f6e6eabdaca3296d5acc2f2bca2"}
Oct 30 21:52:40.808317 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:40.808287    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:40.808437 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:40.808431    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:40.808492 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:40.808455    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kprst" podUID="dbf75704-af7b-44f6-bfcb-b2b2e943f5e6"
Oct 30 21:52:40.808492 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:40.808483    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5h24" podUID="60e220ff-82ff-4d25-b53d-6e98dd4da3fd"
Oct 30 21:52:40.808594 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:40.808519    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:40.808594 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:40.808566    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4tlxp" podUID="4cced475-7943-49ad-870b-4b7119d62759"
Oct 30 21:52:40.970085 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:40.970057    2539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 21:52:40.970605 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:40.970431    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5" event={"ID":"d46f22d3-4503-41af-98d9-c641d343de28","Type":"ContainerStarted","Data":"27175e9d4547cc05eda8166822983d630070d5cb304670f30cd40c94d836cbbd"}
Oct 30 21:52:40.973207 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:40.973181    2539 generic.go:356] "Generic (PLEG): container finished" podID="6aea3c42-943b-4532-b21d-64ada9922ed2" containerID="a42cfd90b73e7c8f9a0b9d664c9382a079ff4ce0ff85dd2348601dadc8d1f3fe" exitCode=0
Oct 30 21:52:40.973330 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:40.973220    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28mbh" event={"ID":"6aea3c42-943b-4532-b21d-64ada9922ed2","Type":"ContainerDied","Data":"a42cfd90b73e7c8f9a0b9d664c9382a079ff4ce0ff85dd2348601dadc8d1f3fe"}
Oct 30 21:52:41.978171 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:41.978135    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28mbh" event={"ID":"6aea3c42-943b-4532-b21d-64ada9922ed2","Type":"ContainerStarted","Data":"12e6067a1c8d220ef22689ce6445433d216230ce8e35e38ef37a6d31cd1e2d14"}
Oct 30 21:52:42.096041 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:42.095994    2539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-28mbh" podStartSLOduration=8.382475522 podStartE2EDuration="21.0959802s" totalImagesPullingTime="12.713504678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:52:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:42.095812841 +0000 UTC m=+21.737702447" watchObservedRunningTime="2026-10-30 21:52:42.0959802 +0000 UTC m=+21.737869821"
Oct 30 21:52:42.810601 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:42.810570    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:42.810767 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:42.810646    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:42.810767 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:42.810732    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:42.810767 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:42.810740    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4tlxp" podUID="4cced475-7943-49ad-870b-4b7119d62759"
Oct 30 21:52:42.810974 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:42.810826    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kprst" podUID="dbf75704-af7b-44f6-bfcb-b2b2e943f5e6"
Oct 30 21:52:42.810974 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:42.810916    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5h24" podUID="60e220ff-82ff-4d25-b53d-6e98dd4da3fd"
Oct 30 21:52:42.983359 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:42.983334    2539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 21:52:42.983998 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:42.983764    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5" event={"ID":"d46f22d3-4503-41af-98d9-c641d343de28","Type":"ContainerStarted","Data":"120b177ced1aff20d1e87412b295efa601e212a9f2ce7ed16a4e7944303de853"}
Oct 30 21:52:42.984328 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:42.984308    2539 scope.go:122] "RemoveContainer" containerID="d663386d48bee40654cd125a6da85b38dd8605fb434953a9ab8a4a084ffafd5c"
Oct 30 21:52:43.989775 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:43.989595    2539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 21:52:43.990302 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:43.990095    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5" event={"ID":"d46f22d3-4503-41af-98d9-c641d343de28","Type":"ContainerStarted","Data":"8f5a2e78534c2f06a9ed659fdd9bb476e83b9573498bdcfce552ea0894cb897d"}
Oct 30 21:52:43.990370 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:43.990359    2539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:43.990427 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:43.990418    2539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:43.990484 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:43.990465    2539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:44.006314 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:44.006287    2539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:44.007434 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:44.007414    2539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:52:44.266153 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:44.266053    2539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5" podStartSLOduration=8.075782019 podStartE2EDuration="23.266035185s" totalImagesPullingTime="15.190253166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:44.216670946 +0000 UTC m=+23.858560550" watchObservedRunningTime="2026-10-30 21:52:44.266035185 +0000 UTC m=+23.907924790"
Oct 30 21:52:44.807542 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:44.807508    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:44.807542 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:44.807521    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:44.807723 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:44.807595    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4tlxp" podUID="4cced475-7943-49ad-870b-4b7119d62759"
Oct 30 21:52:44.807723 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:44.807645    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:44.807815 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:44.807730    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5h24" podUID="60e220ff-82ff-4d25-b53d-6e98dd4da3fd"
Oct 30 21:52:44.807815 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:44.807795    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kprst" podUID="dbf75704-af7b-44f6-bfcb-b2b2e943f5e6"
Oct 30 21:52:45.547456 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:45.547372    2539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4tlxp"]
Oct 30 21:52:45.548015 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:45.547495    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:45.548015 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:45.547610    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4tlxp" podUID="4cced475-7943-49ad-870b-4b7119d62759"
Oct 30 21:52:45.550172 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:45.550145    2539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kprst"]
Oct 30 21:52:45.550295 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:45.550245    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:45.550356 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:45.550336    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kprst" podUID="dbf75704-af7b-44f6-bfcb-b2b2e943f5e6"
Oct 30 21:52:45.550734 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:45.550715    2539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b5h24"]
Oct 30 21:52:45.550828 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:45.550802    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:45.550907 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:45.550890    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5h24" podUID="60e220ff-82ff-4d25-b53d-6e98dd4da3fd"
Oct 30 21:52:46.807255 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:46.807200    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:46.807255 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:46.807241    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:46.807657 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:46.807350    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kprst" podUID="dbf75704-af7b-44f6-bfcb-b2b2e943f5e6"
Oct 30 21:52:46.807657 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:46.807464    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4tlxp" podUID="4cced475-7943-49ad-870b-4b7119d62759"
Oct 30 21:52:47.807300 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:47.807270    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:47.807670 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:47.807368    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5h24" podUID="60e220ff-82ff-4d25-b53d-6e98dd4da3fd"
Oct 30 21:52:48.807589 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:48.807558    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:48.807990 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:48.807685    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:48.807990 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:48.807702    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kprst" podUID="dbf75704-af7b-44f6-bfcb-b2b2e943f5e6"
Oct 30 21:52:48.807990 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:48.807775    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4tlxp" podUID="4cced475-7943-49ad-870b-4b7119d62759"
Oct 30 21:52:49.807632 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:49.807604    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:49.808085 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:49.807732    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5h24" podUID="60e220ff-82ff-4d25-b53d-6e98dd4da3fd"
Oct 30 21:52:50.403174 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:50.403106    2539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:50.628762 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:50.628728    2539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:50.812076 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:50.812000    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:50.812492 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:50.812185    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:50.812492 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:50.812181    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kprst" podUID="dbf75704-af7b-44f6-bfcb-b2b2e943f5e6"
Oct 30 21:52:50.812492 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:50.812263    2539 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4tlxp" podUID="4cced475-7943-49ad-870b-4b7119d62759"
Oct 30 21:52:50.930339 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:50.930101    2539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:51.096798 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.096717    2539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:51.304584 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.304552    2539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:51.600121 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.600097    2539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeReady"
Oct 30 21:52:51.600303 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.600248    2539 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:52:51.712328 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.712257    2539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wx8d9"]
Oct 30 21:52:51.753640 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.753616    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:52:51.761859 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.761833    2539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8w5d6"]
Oct 30 21:52:51.776790 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.776768    2539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8w5d6"]
Oct 30 21:52:51.776890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.776807    2539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wx8d9"]
Oct 30 21:52:51.776890 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.776865    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8w5d6"
Oct 30 21:52:51.807254 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.807233    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:51.807740 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.807720    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:52:51.807843 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.807826    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gsd79\""
Oct 30 21:52:51.807941 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.807927    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:52:51.866464 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.866437    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:52:51.866809 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.866476    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/176bb59e-df7f-4b7a-8335-4c23f633ac1f-cert\") pod \"ingress-canary-8w5d6\" (UID: \"176bb59e-df7f-4b7a-8335-4c23f633ac1f\") " pod="openshift-ingress-canary/ingress-canary-8w5d6"
Oct 30 21:52:51.866809 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.866491    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/18aafbda-138f-40ee-9b40-9e1b76163b98-config-volume\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:52:51.866809 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.866565    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbmhw\" (UniqueName: \"kubernetes.io/projected/176bb59e-df7f-4b7a-8335-4c23f633ac1f-kube-api-access-qbmhw\") pod \"ingress-canary-8w5d6\" (UID: \"176bb59e-df7f-4b7a-8335-4c23f633ac1f\") " pod="openshift-ingress-canary/ingress-canary-8w5d6"
Oct 30 21:52:51.866809 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.866641    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8vbm\" (UniqueName: \"kubernetes.io/projected/18aafbda-138f-40ee-9b40-9e1b76163b98-kube-api-access-h8vbm\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:52:51.866809 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.866669    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18aafbda-138f-40ee-9b40-9e1b76163b98-tmp-dir\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:52:51.907941 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.907911    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:52:51.908107 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.907986    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:52:51.908107 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.908007    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:52:51.908107 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.908009    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:52:51.908107 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.908007    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:52:51.908107 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.908030    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gmfbd\""
Oct 30 21:52:51.908107 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.908103    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-qt896\""
Oct 30 21:52:51.967327 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.967257    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/176bb59e-df7f-4b7a-8335-4c23f633ac1f-cert\") pod \"ingress-canary-8w5d6\" (UID: \"176bb59e-df7f-4b7a-8335-4c23f633ac1f\") " pod="openshift-ingress-canary/ingress-canary-8w5d6"
Oct 30 21:52:51.967327 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.967289    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/18aafbda-138f-40ee-9b40-9e1b76163b98-config-volume\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:52:51.967327 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.967315    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbmhw\" (UniqueName: \"kubernetes.io/projected/176bb59e-df7f-4b7a-8335-4c23f633ac1f-kube-api-access-qbmhw\") pod \"ingress-canary-8w5d6\" (UID: \"176bb59e-df7f-4b7a-8335-4c23f633ac1f\") " pod="openshift-ingress-canary/ingress-canary-8w5d6"
Oct 30 21:52:51.967576 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.967344    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8vbm\" (UniqueName: \"kubernetes.io/projected/18aafbda-138f-40ee-9b40-9e1b76163b98-kube-api-access-h8vbm\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:52:51.967576 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.967367    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18aafbda-138f-40ee-9b40-9e1b76163b98-tmp-dir\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:52:51.967576 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.967409    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:52:51.967576 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:51.967542    2539 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:52:51.967752 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:51.967607    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls podName:18aafbda-138f-40ee-9b40-9e1b76163b98 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:52.467586278 +0000 UTC m=+32.109475865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls") pod "dns-default-wx8d9" (UID: "18aafbda-138f-40ee-9b40-9e1b76163b98") : secret "dns-default-metrics-tls" not found
Oct 30 21:52:51.967752 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.967717    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18aafbda-138f-40ee-9b40-9e1b76163b98-tmp-dir\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:52:51.967945 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.967921    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/18aafbda-138f-40ee-9b40-9e1b76163b98-config-volume\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:52:51.971602 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:51.971582    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/176bb59e-df7f-4b7a-8335-4c23f633ac1f-cert\") pod \"ingress-canary-8w5d6\" (UID: \"176bb59e-df7f-4b7a-8335-4c23f633ac1f\") " pod="openshift-ingress-canary/ingress-canary-8w5d6"
Oct 30 21:52:52.024269 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:52.024232    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbmhw\" (UniqueName: \"kubernetes.io/projected/176bb59e-df7f-4b7a-8335-4c23f633ac1f-kube-api-access-qbmhw\") pod \"ingress-canary-8w5d6\" (UID: \"176bb59e-df7f-4b7a-8335-4c23f633ac1f\") " pod="openshift-ingress-canary/ingress-canary-8w5d6"
Oct 30 21:52:52.024403 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:52.024321    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8vbm\" (UniqueName: \"kubernetes.io/projected/18aafbda-138f-40ee-9b40-9e1b76163b98-kube-api-access-h8vbm\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:52:52.105514 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:52.105477    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8w5d6"
Oct 30 21:52:52.472186 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:52.472151    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:52:52.472346 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:52.472263    2539 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:52:52.472346 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:52.472314    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls podName:18aafbda-138f-40ee-9b40-9e1b76163b98 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:53.472302854 +0000 UTC m=+33.114192436 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls") pod "dns-default-wx8d9" (UID: "18aafbda-138f-40ee-9b40-9e1b76163b98") : secret "dns-default-metrics-tls" not found
Oct 30 21:52:52.519651 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:52.519611    2539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8w5d6"]
Oct 30 21:52:52.526002 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:52.525945    2539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176bb59e_df7f_4b7a_8335_4c23f633ac1f.slice/crio-b14a7da0cb0afcf499ddc111e19b0691cc9ccf85365b730a1adb2f47217754c9 WatchSource:0}: Error finding container b14a7da0cb0afcf499ddc111e19b0691cc9ccf85365b730a1adb2f47217754c9: Status 404 returned error can't find the container with id b14a7da0cb0afcf499ddc111e19b0691cc9ccf85365b730a1adb2f47217754c9
Oct 30 21:52:52.656257 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:52.656219    2539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:52.804938 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:52.804835    2539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:52.808003 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:52.807986    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:52.808113 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:52.808020    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:52.859815 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:52.859792    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:52:52.859934 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:52.859855    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7bphz\""
Oct 30 21:52:52.859934 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:52.859924    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:52:52.860096 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:52.860084    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6q7ft\""
Oct 30 21:52:53.011757 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:53.011728    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8w5d6" event={"ID":"176bb59e-df7f-4b7a-8335-4c23f633ac1f","Type":"ContainerStarted","Data":"b14a7da0cb0afcf499ddc111e19b0691cc9ccf85365b730a1adb2f47217754c9"}
Oct 30 21:52:53.479659 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:53.479625    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:52:53.479841 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:53.479737    2539 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:52:53.479841 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:53.479787    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls podName:18aafbda-138f-40ee-9b40-9e1b76163b98 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:55.479775276 +0000 UTC m=+35.121664862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls") pod "dns-default-wx8d9" (UID: "18aafbda-138f-40ee-9b40-9e1b76163b98") : secret "dns-default-metrics-tls" not found
Oct 30 21:52:54.690215 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:54.690177    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:52:54.690803 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:54.690231    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:54.690803 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:54.690364    2539 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:52:54.690803 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:54.690566    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs podName:dbf75704-af7b-44f6-bfcb-b2b2e943f5e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:26.690509726 +0000 UTC m=+66.332399310 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs") pod "network-metrics-daemon-kprst" (UID: "dbf75704-af7b-44f6-bfcb-b2b2e943f5e6") : secret "metrics-daemon-secret" not found
Oct 30 21:52:54.692864 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:54.692843    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:54.790784 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:54.790737    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5zlk\" (UniqueName: \"kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk\") pod \"network-check-target-b5h24\" (UID: \"60e220ff-82ff-4d25-b53d-6e98dd4da3fd\") " pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:54.793524 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:54.793493    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5zlk\" (UniqueName: \"kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk\") pod \"network-check-target-b5h24\" (UID: \"60e220ff-82ff-4d25-b53d-6e98dd4da3fd\") " pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:54.811114 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:54.811086    2539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:54.827958 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:54.827929    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:52:54.951601 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:54.951569    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:52:54.976607 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:54.976582    2539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:55.236883 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:55.236809    2539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b5h24"]
Oct 30 21:52:55.240378 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:55.240342    2539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e220ff_82ff_4d25_b53d_6e98dd4da3fd.slice/crio-087d0377ddb4aa2adfc070dddbd131183fe85e9449af195f4b7874b6b96767f2 WatchSource:0}: Error finding container 087d0377ddb4aa2adfc070dddbd131183fe85e9449af195f4b7874b6b96767f2: Status 404 returned error can't find the container with id 087d0377ddb4aa2adfc070dddbd131183fe85e9449af195f4b7874b6b96767f2
Oct 30 21:52:55.345030 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:55.344997    2539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4tlxp"]
Oct 30 21:52:55.349129 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:55.349093    2539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cced475_7943_49ad_870b_4b7119d62759.slice/crio-fabae2396d46b15dab6233392969f273acac4e110a5489f18714fab2b4888bde WatchSource:0}: Error finding container fabae2396d46b15dab6233392969f273acac4e110a5489f18714fab2b4888bde: Status 404 returned error can't find the container with id fabae2396d46b15dab6233392969f273acac4e110a5489f18714fab2b4888bde
Oct 30 21:52:55.496506 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:55.496434    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:52:55.496639 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:55.496556    2539 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:52:55.496639 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:55.496608    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls podName:18aafbda-138f-40ee-9b40-9e1b76163b98 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:59.496596085 +0000 UTC m=+39.138485668 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls") pod "dns-default-wx8d9" (UID: "18aafbda-138f-40ee-9b40-9e1b76163b98") : secret "dns-default-metrics-tls" not found
Oct 30 21:52:56.021001 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:56.020964    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4tlxp" event={"ID":"4cced475-7943-49ad-870b-4b7119d62759","Type":"ContainerStarted","Data":"fabae2396d46b15dab6233392969f273acac4e110a5489f18714fab2b4888bde"}
Oct 30 21:52:56.022137 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:56.022108    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b5h24" event={"ID":"60e220ff-82ff-4d25-b53d-6e98dd4da3fd","Type":"ContainerStarted","Data":"087d0377ddb4aa2adfc070dddbd131183fe85e9449af195f4b7874b6b96767f2"}
Oct 30 21:52:56.023609 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:56.023589    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8w5d6" event={"ID":"176bb59e-df7f-4b7a-8335-4c23f633ac1f","Type":"ContainerStarted","Data":"3329485ada5796cc0b4ea9234d82d0f0ab4f67fa82eb735cf2d9dae1fd7b2346"}
Oct 30 21:52:56.139390 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:56.139282    2539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8w5d6" podStartSLOduration=1.849198927 podStartE2EDuration="5.139267382s" totalImagesPullingTime="3.290068455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:56.139133697 +0000 UTC m=+35.781023316" watchObservedRunningTime="2026-10-30 21:52:56.139267382 +0000 UTC m=+35.781156987"
Oct 30 21:52:57.871706 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:57.871670    2539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nf7t4"]
Oct 30 21:52:57.894328 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:57.894294    2539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:57.902516 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:57.902491    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:57.938524 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:57.938451    2539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nf7t4"]
Oct 30 21:52:57.986889 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:57.986861    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:52:57.986889 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:57.986861    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h6mh8\""
Oct 30 21:52:58.014583 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.014453    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:58.014583 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.014508    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:58.014583 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.014543    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4d48390-828e-4c3e-91e8-44264a171fd3-crio-socket\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:58.014869 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.014665    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lfjn\" (UniqueName: \"kubernetes.io/projected/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-api-access-4lfjn\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:58.014869 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.014706    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4d48390-828e-4c3e-91e8-44264a171fd3-data-volume\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:58.039089 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.039060    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:52:58.039089 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.039094    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:52:58.039298 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.039108    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:52:58.052790 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.052757    2539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:58.075694 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.075664    2539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-dlnp8"]
Oct 30 21:52:58.096326 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.096267    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dlnp8"
Oct 30 21:52:58.115908 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.115883    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:58.116058 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.115919    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4d48390-828e-4c3e-91e8-44264a171fd3-crio-socket\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:58.116058 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.115982    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lfjn\" (UniqueName: \"kubernetes.io/projected/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-api-access-4lfjn\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:58.116058 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.116004    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4d48390-828e-4c3e-91e8-44264a171fd3-data-volume\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:58.116058 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:58.116029    2539 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:52:58.116058 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.116040    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:58.116285 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:58.116104    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-rbac-proxy-cm podName:b4d48390-828e-4c3e-91e8-44264a171fd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:58.616084123 +0000 UTC m=+38.257973708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nf7t4" (UID: "b4d48390-828e-4c3e-91e8-44264a171fd3") : configmap "kube-rbac-proxy" not found
Oct 30 21:52:58.116285 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:58.116136    2539 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:52:58.116285 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:58.116185    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls podName:b4d48390-828e-4c3e-91e8-44264a171fd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:58.616170389 +0000 UTC m=+38.258060000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nf7t4" (UID: "b4d48390-828e-4c3e-91e8-44264a171fd3") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:52:58.116285 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.116222    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4d48390-828e-4c3e-91e8-44264a171fd3-crio-socket\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:58.116506 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.116406    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4d48390-828e-4c3e-91e8-44264a171fd3-data-volume\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:58.127827 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.127760    2539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-w68zw-mzlkv"]
Oct 30 21:52:58.148725 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.148672    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:52:58.148725 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.148691    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:52:58.148929 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.148811    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jfz4q\""
Oct 30 21:52:58.150668 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.150620    2539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-dlnp8"]
Oct 30 21:52:58.150794 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.150717    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:52:58.172773 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.172747    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lfjn\" (UniqueName: \"kubernetes.io/projected/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-api-access-4lfjn\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:58.187981 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.187935    2539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-w68zw-mzlkv"]
Oct 30 21:52:58.197698 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.197666    2539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:58.216343 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.216315    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8faee8d2-4d27-4709-af52-4b5ef32592f6-archives-path\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:52:58.216525 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.216362    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-service-ca-bundle\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:52:58.216525 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.216404    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-trusted-ca-bundle\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:52:58.216525 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.216460    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m744c\" (UniqueName: \"kubernetes.io/projected/8faee8d2-4d27-4709-af52-4b5ef32592f6-kube-api-access-m744c\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:52:58.216525 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.216495    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49qz4\" (UniqueName: \"kubernetes.io/projected/f37fd220-ad2c-4fd0-9d24-6b9733ce7d28-kube-api-access-49qz4\") pod \"migrator-66df5d496b-dlnp8\" (UID: \"f37fd220-ad2c-4fd0-9d24-6b9733ce7d28\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dlnp8"
Oct 30 21:52:58.274980 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.274932    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:52:58.275163 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.275130    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jrvhp\""
Oct 30 21:52:58.275357 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.275338    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:52:58.317223 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.317191    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-trusted-ca-bundle\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:52:58.317385 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.317253    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m744c\" (UniqueName: \"kubernetes.io/projected/8faee8d2-4d27-4709-af52-4b5ef32592f6-kube-api-access-m744c\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:52:58.317385 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.317288    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49qz4\" (UniqueName: \"kubernetes.io/projected/f37fd220-ad2c-4fd0-9d24-6b9733ce7d28-kube-api-access-49qz4\") pod \"migrator-66df5d496b-dlnp8\" (UID: \"f37fd220-ad2c-4fd0-9d24-6b9733ce7d28\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dlnp8"
Oct 30 21:52:58.317385 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.317338    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8faee8d2-4d27-4709-af52-4b5ef32592f6-archives-path\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:52:58.317548 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.317382    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-service-ca-bundle\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:52:58.318097 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.318058    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8faee8d2-4d27-4709-af52-4b5ef32592f6-archives-path\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:52:58.318237 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.318115    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-service-ca-bundle\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:52:58.318781 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.318746    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-trusted-ca-bundle\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:52:58.357438 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.357405    2539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:58.375519 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.375487    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m744c\" (UniqueName: \"kubernetes.io/projected/8faee8d2-4d27-4709-af52-4b5ef32592f6-kube-api-access-m744c\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:52:58.376307 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.376262    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49qz4\" (UniqueName: \"kubernetes.io/projected/f37fd220-ad2c-4fd0-9d24-6b9733ce7d28-kube-api-access-49qz4\") pod \"migrator-66df5d496b-dlnp8\" (UID: \"f37fd220-ad2c-4fd0-9d24-6b9733ce7d28\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dlnp8"
Oct 30 21:52:58.429406 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.429368    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dlnp8"
Oct 30 21:52:58.473548 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.473517    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:52:58.515500 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.514899    2539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:58.620024 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.619993    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:58.620024 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.620033    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:58.620251 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:58.620120    2539 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:52:58.620251 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:58.620174    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-rbac-proxy-cm podName:b4d48390-828e-4c3e-91e8-44264a171fd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:59.620156075 +0000 UTC m=+39.262045663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nf7t4" (UID: "b4d48390-828e-4c3e-91e8-44264a171fd3") : configmap "kube-rbac-proxy" not found
Oct 30 21:52:58.620528 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:58.620511    2539 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:52:58.620582 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:58.620561    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls podName:b4d48390-828e-4c3e-91e8-44264a171fd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:59.620550566 +0000 UTC m=+39.262440149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nf7t4" (UID: "b4d48390-828e-4c3e-91e8-44264a171fd3") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:52:58.845160 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.845054    2539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-dlnp8"]
Oct 30 21:52:58.884556 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:58.884528    2539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf37fd220_ad2c_4fd0_9d24_6b9733ce7d28.slice/crio-c5d48248be9a0e8dda3d52f5736448c3bb5c10fcc9c17e15ddbeee3a0d19347b WatchSource:0}: Error finding container c5d48248be9a0e8dda3d52f5736448c3bb5c10fcc9c17e15ddbeee3a0d19347b: Status 404 returned error can't find the container with id c5d48248be9a0e8dda3d52f5736448c3bb5c10fcc9c17e15ddbeee3a0d19347b
Oct 30 21:52:58.894534 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:58.894508    2539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-w68zw-mzlkv"]
Oct 30 21:52:58.898666 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:52:58.898643    2539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8faee8d2_4d27_4709_af52_4b5ef32592f6.slice/crio-6d3fe2f59da01374bc7947e14cfb67ac02039b9d7a0ba6a6036cee13bb2d3b6f WatchSource:0}: Error finding container 6d3fe2f59da01374bc7947e14cfb67ac02039b9d7a0ba6a6036cee13bb2d3b6f: Status 404 returned error can't find the container with id 6d3fe2f59da01374bc7947e14cfb67ac02039b9d7a0ba6a6036cee13bb2d3b6f
Oct 30 21:52:59.033256 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.033217    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dlnp8" event={"ID":"f37fd220-ad2c-4fd0-9d24-6b9733ce7d28","Type":"ContainerStarted","Data":"c5d48248be9a0e8dda3d52f5736448c3bb5c10fcc9c17e15ddbeee3a0d19347b"}
Oct 30 21:52:59.035050 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.035008    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w68zw-mzlkv" event={"ID":"8faee8d2-4d27-4709-af52-4b5ef32592f6","Type":"ContainerStarted","Data":"6d3fe2f59da01374bc7947e14cfb67ac02039b9d7a0ba6a6036cee13bb2d3b6f"}
Oct 30 21:52:59.261504 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.261467    2539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-72s7t"]
Oct 30 21:52:59.282157 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.282126    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t"
Oct 30 21:52:59.322372 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.322337    2539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-72s7t"]
Oct 30 21:52:59.334061 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.333936    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 21:52:59.334061 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.334026    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 21:52:59.334245 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.334067    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 21:52:59.334245 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.334078    2539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fmdw6\""
Oct 30 21:52:59.334245 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.334067    2539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 21:52:59.427526 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.427486    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-signing-cabundle\") pod \"service-ca-5fffc97bdf-72s7t\" (UID: \"2aa2d0cb-19e6-4d3a-8aae-7760c255bef3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t"
Oct 30 21:52:59.427526 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.427521    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-signing-key\") pod \"service-ca-5fffc97bdf-72s7t\" (UID: \"2aa2d0cb-19e6-4d3a-8aae-7760c255bef3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t"
Oct 30 21:52:59.427743 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.427708    2539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wqh2\" (UniqueName: \"kubernetes.io/projected/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-kube-api-access-5wqh2\") pod \"service-ca-5fffc97bdf-72s7t\" (UID: \"2aa2d0cb-19e6-4d3a-8aae-7760c255bef3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t"
Oct 30 21:52:59.528174 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.528131    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:52:59.528334 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.528181    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wqh2\" (UniqueName: \"kubernetes.io/projected/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-kube-api-access-5wqh2\") pod \"service-ca-5fffc97bdf-72s7t\" (UID: \"2aa2d0cb-19e6-4d3a-8aae-7760c255bef3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t"
Oct 30 21:52:59.528334 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.528225    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-signing-cabundle\") pod \"service-ca-5fffc97bdf-72s7t\" (UID: \"2aa2d0cb-19e6-4d3a-8aae-7760c255bef3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t"
Oct 30 21:52:59.528334 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.528253    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-signing-key\") pod \"service-ca-5fffc97bdf-72s7t\" (UID: \"2aa2d0cb-19e6-4d3a-8aae-7760c255bef3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t"
Oct 30 21:52:59.528334 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:59.528283    2539 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:52:59.528542 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:59.528356    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls podName:18aafbda-138f-40ee-9b40-9e1b76163b98 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:07.52833585 +0000 UTC m=+47.170225449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls") pod "dns-default-wx8d9" (UID: "18aafbda-138f-40ee-9b40-9e1b76163b98") : secret "dns-default-metrics-tls" not found
Oct 30 21:52:59.528969 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.528925    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-signing-cabundle\") pod \"service-ca-5fffc97bdf-72s7t\" (UID: \"2aa2d0cb-19e6-4d3a-8aae-7760c255bef3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t"
Oct 30 21:52:59.531120 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.531101    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-signing-key\") pod \"service-ca-5fffc97bdf-72s7t\" (UID: \"2aa2d0cb-19e6-4d3a-8aae-7760c255bef3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t"
Oct 30 21:52:59.588679 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.588638    2539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wqh2\" (UniqueName: \"kubernetes.io/projected/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-kube-api-access-5wqh2\") pod \"service-ca-5fffc97bdf-72s7t\" (UID: \"2aa2d0cb-19e6-4d3a-8aae-7760c255bef3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t"
Oct 30 21:52:59.608475 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.608446    2539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t"
Oct 30 21:52:59.628799 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.628776    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:59.628921 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:59.628881    2539 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:52:59.628996 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:59.628973    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-rbac-proxy-cm podName:b4d48390-828e-4c3e-91e8-44264a171fd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:01.628934409 +0000 UTC m=+41.270824008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nf7t4" (UID: "b4d48390-828e-4c3e-91e8-44264a171fd3") : configmap "kube-rbac-proxy" not found
Oct 30 21:52:59.628996 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:59.628983    2539 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:52:59.629086 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:52:59.629028    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls podName:b4d48390-828e-4c3e-91e8-44264a171fd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:01.629017121 +0000 UTC m=+41.270906704 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nf7t4" (UID: "b4d48390-828e-4c3e-91e8-44264a171fd3") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:52:59.629086 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.628889    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:52:59.640857 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.640827    2539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:59.826478 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:52:59.826435    2539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:00.002464 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:53:00.002392    2539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-72s7t"]
Oct 30 21:53:00.007462 ip-10-0-130-162 kubenswrapper[2539]: W1030 21:53:00.007431    2539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa2d0cb_19e6_4d3a_8aae_7760c255bef3.slice/crio-bb3171025f33b95209e823494f690f70835c3e48141ef8a53b01609108e16607 WatchSource:0}: Error finding container bb3171025f33b95209e823494f690f70835c3e48141ef8a53b01609108e16607: Status 404 returned error can't find the container with id bb3171025f33b95209e823494f690f70835c3e48141ef8a53b01609108e16607
Oct 30 21:53:00.015227 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:53:00.015200    2539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:00.040533 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:53:00.040419    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b5h24" event={"ID":"60e220ff-82ff-4d25-b53d-6e98dd4da3fd","Type":"ContainerStarted","Data":"ee3dfffe05fbf71822ff9d3b99121579206fdb9909392afae4a5f59fa2e57e20"}
Oct 30 21:53:00.040533 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:53:00.040501    2539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:53:00.042012 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:53:00.041989    2539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t" event={"ID":"2aa2d0cb-19e6-4d3a-8aae-7760c255bef3","Type":"ContainerStarted","Data":"bb3171025f33b95209e823494f690f70835c3e48141ef8a53b01609108e16607"}
Oct 30 21:53:00.155465 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:53:00.155402    2539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b5h24" podStartSLOduration=35.327655957 podStartE2EDuration="39.155383494s" totalImagesPullingTime="3.827727537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:00.154546899 +0000 UTC m=+39.796436527" watchObservedRunningTime="2026-10-30 21:53:00.155383494 +0000 UTC m=+39.797273100"
Oct 30 21:53:01.645689 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:53:01.645654    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:01.646115 ip-10-0-130-162 kubenswrapper[2539]: I1030 21:53:01.645772    2539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:01.646115 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:53:01.645793    2539 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:53:01.646115 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:53:01.645874    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-rbac-proxy-cm podName:b4d48390-828e-4c3e-91e8-44264a171fd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:05.645853717 +0000 UTC m=+45.287743301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nf7t4" (UID: "b4d48390-828e-4c3e-91e8-44264a171fd3") : configmap "kube-rbac-proxy" not found
Oct 30 21:53:01.646115 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:53:01.645896    2539 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:53:01.646115 ip-10-0-130-162 kubenswrapper[2539]: E1030 21:53:01.645976    2539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls podName:b4d48390-828e-4c3e-91e8-44264a171fd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:05.645935801 +0000 UTC m=+45.287825405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nf7t4" (UID: "b4d48390-828e-4c3e-91e8-44264a171fd3") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:53:01.682737 ip-10-0-130-162 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 21:53:01.693451 ip-10-0-130-162 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 21:53:01.693688 ip-10-0-130-162 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 21:53:01.693749 ip-10-0-130-162 systemd[1]: kubelet.service: Consumed 2.478s CPU time.
Oct 30 21:53:01.704407 ip-10-0-130-162 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:53:01.793464 ip-10-0-130-162 kubenswrapper[4473]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:53:01.793464 ip-10-0-130-162 kubenswrapper[4473]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:53:01.793464 ip-10-0-130-162 kubenswrapper[4473]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:53:01.793464 ip-10-0-130-162 kubenswrapper[4473]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:53:01.795509 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795424    4473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:53:01.795509 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795510    4473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:53:01.795509 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795514    4473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795517    4473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795520    4473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795523    4473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795525    4473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795528    4473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795531    4473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795533    4473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795536    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795538    4473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795541    4473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795544    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795546    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795549    4473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795551    4473 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795553    4473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795556    4473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795558    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795561    4473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:53:01.795650 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795564    4473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795566    4473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795569    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795571    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795574    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795578    4473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795580    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795583    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795586    4473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795589    4473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795591    4473 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795594    4473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795597    4473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795599    4473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795602    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795604    4473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795607    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795609    4473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795612    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:53:01.796141 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795614    4473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795616    4473 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795619    4473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795624    4473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795629    4473 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795632    4473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795635    4473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795638    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795642    4473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795646    4473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795649    4473 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795652    4473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795655    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795658    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795660    4473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795663    4473 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795666    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795668    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795671    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795673    4473 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:53:01.796602 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795676    4473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795678    4473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795682    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795685    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795687    4473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795690    4473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795692    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795695    4473 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795698    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795700    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795704    4473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795706    4473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795709    4473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795711    4473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795714    4473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795717    4473 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795720    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795722    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795725    4473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:53:01.797109 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795727    4473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795730    4473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795733    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795735    4473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795738    4473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795740    4473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795743    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795745    4473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795748    4473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795750    4473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795753    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795755    4473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795759    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795761    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795764    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795767    4473 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795769    4473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795772    4473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795774    4473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795777    4473 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:53:01.797566 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795779    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795782    4473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795784    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795787    4473 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795789    4473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795792    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.795794    4473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796547    4473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796554    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796557    4473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796559    4473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796562    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796564    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796567    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796570    4473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796573    4473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796576    4473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796579    4473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796582    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:53:01.798059 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796584    4473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796587    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796590    4473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796593    4473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796595    4473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796599    4473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796603    4473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796606    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796608    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796611    4473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796614    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796617    4473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796619    4473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796622    4473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796625    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796627    4473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796629    4473 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796632    4473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796634    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:53:01.798525 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796637    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796639    4473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796641    4473 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796646    4473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796649    4473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796654    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796657    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796660    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796663    4473 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796666    4473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796668    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796671    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796674    4473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796676    4473 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796679    4473 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796682    4473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796684    4473 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796687    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796689    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:53:01.798997 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796692    4473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796695    4473 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796697    4473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796700    4473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796702    4473 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796705    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796708    4473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796710    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796713    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796715    4473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796718    4473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796720    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796723    4473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796725    4473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796728    4473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796730    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796732    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796735    4473 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796738    4473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796741    4473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:53:01.799461 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796743    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796746    4473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796749    4473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796752    4473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796754    4473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796757    4473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796759    4473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796762    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796764    4473 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796767    4473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796769    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796772    4473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796774    4473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796776    4473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796779    4473 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796781    4473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796784    4473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796786    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796789    4473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796791    4473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:53:01.799931 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796794    4473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796796    4473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796798    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796801    4473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796803    4473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796806    4473 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796808    4473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796811    4473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796813    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796815    4473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796818    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796821    4473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796823    4473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796826    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796829    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.796832    4473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797313    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797318    4473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797320    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797323    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:53:01.800425 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797326    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797329    4473 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797331    4473 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797334    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797336    4473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797346    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797348    4473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797351    4473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797353    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797356    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797358    4473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797362    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797364    4473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797367    4473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797369    4473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797372    4473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797374    4473 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797377    4473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797380    4473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797383    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:53:01.800938 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797385    4473 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797388    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797391    4473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797393    4473 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797396    4473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797398    4473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797401    4473 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797404    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797407    4473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797409    4473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797412    4473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797414    4473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797417    4473 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797419    4473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797422    4473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797424    4473 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797426    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797431    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797433    4473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797436    4473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:53:01.801428 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797438    4473 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797441    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797443    4473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797446    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797449    4473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797451    4473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797454    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797456    4473 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797462    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797465    4473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797467    4473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797470    4473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797473    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797476    4473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797478    4473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797481    4473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797483    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797486    4473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797488    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:53:01.802065 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797491    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797494    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797497    4473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797499    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797502    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797504    4473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797507    4473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797509    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797512    4473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797514    4473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797518    4473 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797521    4473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797523    4473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797527    4473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797530    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797533    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797535    4473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797538    4473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797540    4473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:53:01.802757 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797543    4473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797545    4473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797549    4473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797551    4473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797554    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797557    4473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797559    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797562    4473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797566    4473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797570    4473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797573    4473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797576    4473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797579    4473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797581    4473 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797584    4473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797587    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797589    4473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797593    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797595    4473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:53:01.803254 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797598    4473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797600    4473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797603    4473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797605    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.797609    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797715    4473 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797735    4473 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797742    4473 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797747    4473 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797752    4473 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797756    4473 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797760    4473 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797765    4473 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797768    4473 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797771    4473 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797776    4473 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797779    4473 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797782    4473 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797785    4473 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797788    4473 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797791    4473 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797793    4473 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797796    4473 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:53:01.803832 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797801    4473 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797804    4473 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797807    4473 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797810    4473 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797813    4473 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797817    4473 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797821    4473 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797824    4473 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797827    4473 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797830    4473 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797833    4473 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797836    4473 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797839    4473 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797843    4473 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797848    4473 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797851    4473 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797854    4473 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797856    4473 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797859    4473 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797862    4473 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797866    4473 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797870    4473 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797873    4473 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797876    4473 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:53:01.804399 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797880    4473 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797885    4473 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797888    4473 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797891    4473 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797894    4473 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797897    4473 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797900    4473 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797903    4473 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797906    4473 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797908    4473 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797911    4473 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797914    4473 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797918    4473 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797921    4473 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797924    4473 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797928    4473 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797931    4473 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797934    4473 flags.go:64] FLAG: --help="false"
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797936    4473 flags.go:64] FLAG: --hostname-override="ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797940    4473 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797943    4473 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797962    4473 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797965    4473 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:53:01.805006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797969    4473 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797972    4473 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797975    4473 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797978    4473 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797981    4473 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797984    4473 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797987    4473 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797990    4473 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797993    4473 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.797998    4473 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798001    4473 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798004    4473 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798006    4473 flags.go:64] FLAG: --lock-file=""
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798009    4473 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798012    4473 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798016    4473 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798024    4473 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798029    4473 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798034    4473 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798038    4473 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798042    4473 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798047    4473 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798052    4473 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798054    4473 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798059    4473 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:53:01.805556 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798063    4473 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798067    4473 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798070    4473 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798073    4473 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798076    4473 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798081    4473 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798084    4473 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798086    4473 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798089    4473 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798096    4473 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798099    4473 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798102    4473 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798105    4473 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798108    4473 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798111    4473 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798113    4473 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798118    4473 flags.go:64] FLAG: --port="10250"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798120    4473 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798123    4473 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0fdf70df224af0e3b"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798126    4473 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798129    4473 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798132    4473 flags.go:64] FLAG: --register-node="true"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798135    4473 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798139    4473 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798142    4473 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798144    4473 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:53:01.806198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798147    4473 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798151    4473 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798154    4473 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798157    4473 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798160    4473 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798163    4473 flags.go:64] FLAG: --runonce="false"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798166    4473 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798170    4473 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798173    4473 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798176    4473 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798179    4473 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798183    4473 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798186    4473 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798189    4473 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798192    4473 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798195    4473 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798198    4473 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798200    4473 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798204    4473 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798206    4473 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798209    4473 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798214    4473 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798218    4473 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798220    4473 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798225    4473 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:53:01.806833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798228    4473 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798231    4473 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798234    4473 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798236    4473 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798239    4473 flags.go:64] FLAG: --v="2"
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798244    4473 flags.go:64] FLAG: --version="false"
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798248    4473 flags.go:64] FLAG: --vmodule=""
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798252    4473 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798255    4473 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798393    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798397    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798399    4473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798402    4473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798404    4473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798407    4473 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798410    4473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798412    4473 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798415    4473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798419    4473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798422    4473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798425    4473 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798427    4473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:53:01.807541 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798430    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798432    4473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798435    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798437    4473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798440    4473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798442    4473 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798445    4473 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798449    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798451    4473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798454    4473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798456    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798459    4473 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798461    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798464    4473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798467    4473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798469    4473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798472    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798474    4473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798477    4473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798479    4473 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:53:01.808183 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798482    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798484    4473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798487    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798489    4473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798492    4473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798494    4473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798498    4473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798502    4473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798506    4473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798510    4473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798512    4473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798515    4473 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798518    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798520    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798523    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798525    4473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798528    4473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798530    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798533    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:53:01.808933 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798536    4473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798539    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798543    4473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798547    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798550    4473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798552    4473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798555    4473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798558    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798561    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798563    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798566    4473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798568    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798571    4473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798573    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798576    4473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798578    4473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798580    4473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798583    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798585    4473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:53:01.809487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798588    4473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798591    4473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798594    4473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798597    4473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798600    4473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798602    4473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798604    4473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798607    4473 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798609    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798612    4473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798614    4473 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798617    4473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798620    4473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798622    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798625    4473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798627    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798630    4473 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798632    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798635    4473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798637    4473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:53:01.810044 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798640    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:53:01.810569 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798642    4473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:53:01.810569 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798645    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:53:01.810569 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798647    4473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:53:01.810569 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798649    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:53:01.810569 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798652    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:53:01.810569 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798654    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:53:01.810569 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798657    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:53:01.810569 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798659    4473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:53:01.810569 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798662    4473 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:53:01.810569 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798664    4473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:53:01.810569 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798666    4473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:53:01.810569 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798669    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:53:01.810569 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798671    4473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:53:01.810569 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798676    4473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:53:01.810569 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.798702    4473 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:53:01.810569 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798763    4473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798769    4473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798772    4473 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798775    4473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798777    4473 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798780    4473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798796    4473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798799    4473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798802    4473 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798804    4473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798807    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798809    4473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798812    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798814    4473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798817    4473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798819    4473 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798822    4473 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798824    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798827    4473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798829    4473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:53:01.810993 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798831    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798834    4473 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798836    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798839    4473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798842    4473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798844    4473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798847    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798849    4473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798851    4473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798854    4473 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798858    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798861    4473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798864    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798866    4473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798869    4473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798872    4473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798876    4473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798879    4473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798882    4473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:53:01.811607 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798885    4473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798888    4473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798890    4473 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798893    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798896    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798898    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798901    4473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798903    4473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798905    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798908    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798910    4473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798913    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798916    4473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798920    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798922    4473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798925    4473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798927    4473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798930    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798932    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:53:01.812218 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798935    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798937    4473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798939    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798942    4473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798946    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798962    4473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798965    4473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798967    4473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798971    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798974    4473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798977    4473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798979    4473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798982    4473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798984    4473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.798987    4473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799000    4473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799004    4473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799007    4473 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799010    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799012    4473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:53:01.812763 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799015    4473 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799017    4473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799020    4473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799022    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799025    4473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799028    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799030    4473 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799032    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799035    4473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799037    4473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799040    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799042    4473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799045    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799047    4473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799050    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799052    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799055    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799058    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799060    4473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:53:01.813266 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799063    4473 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799068    4473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799070    4473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799073    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799076    4473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799078    4473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799081    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799084    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.799087    4473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.803913    4473 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.803930    4473 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.803963    4473 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.803969    4473 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804028    4473 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804033    4473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804036    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804039    4473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804042    4473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804045    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804048    4473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:53:01.813741 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804051    4473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804054    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804057    4473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804060    4473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804062    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804065    4473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804068    4473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804071    4473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804073    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804076    4473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804078    4473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804080    4473 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804083    4473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804086    4473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804089    4473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804092    4473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804094    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804097    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804100    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:53:01.814231 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804104    4473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804109    4473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804113    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804116    4473 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804119    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804121    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804124    4473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804127    4473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804129    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804132    4473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804134    4473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804137    4473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804139    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804142    4473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804144    4473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804147    4473 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804149    4473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804152    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804154    4473 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804157    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:53:01.814695 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804159    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804162    4473 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804164    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804166    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804169    4473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804171    4473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804174    4473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804177    4473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804179    4473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804182    4473 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804184    4473 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804187    4473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804189    4473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804193    4473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804198    4473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804201    4473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804204    4473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804206    4473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804209    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804212    4473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:53:01.815248 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804214    4473 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804217    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804219    4473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804222    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804224    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804227    4473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804229    4473 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804232    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804234    4473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804237    4473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804239    4473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804242    4473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804245    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804248    4473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804251    4473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804253    4473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804256    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804258    4473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804261    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804264    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:53:01.815718 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804266    4473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804269    4473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804271    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804274    4473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804276    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804278    4473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804281    4473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804283    4473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804286    4473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804289    4473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804291    4473 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804294    4473 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804296    4473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804299    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804302    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804304    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804307    4473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804309    4473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804312    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:53:01.816242 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804315    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:53:01.816700 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.804340    4473 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:53:01.816700 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804492    4473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:53:01.816700 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804497    4473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:53:01.816700 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804501    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:53:01.816700 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804503    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:53:01.816700 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804506    4473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:53:01.816700 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804509    4473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:53:01.816700 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804512    4473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:53:01.816700 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804515    4473 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:53:01.816700 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804517    4473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:53:01.816700 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804520    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:53:01.816700 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804522    4473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:53:01.816700 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804525    4473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:53:01.816700 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804528    4473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:53:01.816700 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804530    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804533    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804536    4473 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804538    4473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804541    4473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804543    4473 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804546    4473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804548    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804551    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804553    4473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804556    4473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804559    4473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804561    4473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804564    4473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804567    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804570    4473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804572    4473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804575    4473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804577    4473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804581    4473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:53:01.817532 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804583    4473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804586    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804588    4473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804590    4473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804593    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804596    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804599    4473 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804601    4473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804604    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804606    4473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804609    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804611    4473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804613    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804616    4473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804618    4473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804621    4473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804623    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804626    4473 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804628    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804630    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:53:01.818314 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804633    4473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804635    4473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804638    4473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804641    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804643    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804646    4473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804649    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804651    4473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804654    4473 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804656    4473 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804659    4473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804661    4473 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804664    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804667    4473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804669    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804672    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804674    4473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804677    4473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804680    4473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804683    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:53:01.818799 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804685    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804688    4473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804691    4473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804693    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804696    4473 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804698    4473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804702    4473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804705    4473 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804708    4473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804711    4473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804714    4473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804716    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804718    4473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804721    4473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804724    4473 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804727    4473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804729    4473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804732    4473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804734    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:53:01.819296 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804737    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:53:01.819740 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804739    4473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:53:01.819740 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804742    4473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:53:01.819740 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804744    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:53:01.819740 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804747    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:53:01.819740 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804749    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:53:01.819740 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804752    4473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:53:01.819740 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804754    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:53:01.819740 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804757    4473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:53:01.819740 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804759    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:53:01.819740 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804762    4473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:53:01.819740 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804764    4473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:53:01.819740 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804767    4473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:53:01.819740 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804770    4473 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:53:01.819740 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.804793    4473 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:53:01.819740 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804825    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:53:01.819740 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804829    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804833    4473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804836    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804839    4473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804842    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804844    4473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804847    4473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804849    4473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804852    4473 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804856    4473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804860    4473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804862    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804865    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804868    4473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804871    4473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804873    4473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804876    4473 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804878    4473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804881    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:53:01.820142 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804884    4473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804886    4473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804889    4473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804891    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804894    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804897    4473 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804900    4473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804902    4473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804905    4473 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804907    4473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804910    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804913    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804915    4473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804918    4473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804921    4473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804923    4473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804925    4473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804928    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804930    4473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804933    4473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:53:01.820823 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804935    4473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804938    4473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804940    4473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804943    4473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804945    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804966    4473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804969    4473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804972    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804974    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804977    4473 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804979    4473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804982    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804984    4473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804990    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804993    4473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804995    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.804998    4473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805001    4473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805003    4473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805006    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:53:01.821661 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805008    4473 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805011    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805014    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805017    4473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805020    4473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805023    4473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805026    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805028    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805031    4473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805033    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805036    4473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805038    4473 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805041    4473 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805043    4473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805046    4473 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805048    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805051    4473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805054    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805056    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805058    4473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:53:01.822520 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805061    4473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805063    4473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805066    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805069    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805072    4473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805076    4473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805079    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805081    4473 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805084    4473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805088    4473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805091    4473 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805094    4473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805097    4473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805100    4473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805103    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805105    4473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805109    4473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805112    4473 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805115    4473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805117    4473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:53:01.823432 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805120    4473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:53:01.824479 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805122    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:53:01.824479 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805126    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:53:01.824479 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805128    4473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:53:01.824479 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805131    4473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:53:01.824479 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:01.805133    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:53:01.824479 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.805192    4473 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:53:01.824479 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.806172    4473 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 21:53:01.824479 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.806229    4473 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 21:53:01.824479 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.806406    4473 server.go:1014] "Starting client certificate rotation"
Oct 30 21:53:01.824479 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.806545    4473 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:53:01.824479 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.806594    4473 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:47:21 +0000 UTC" deadline="2028-07-18 17:43:42.098610566 +0000 UTC"
Oct 30 21:53:01.824479 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.806621    4473 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15043h50m40.29199119s"
Oct 30 21:53:01.824479 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.806890    4473 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:53:01.824479 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.807923    4473 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:53:01.824479 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.809635    4473 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:53:01.824479 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.811190    4473 log.go:25] "Validated CRI v1 image API"
Oct 30 21:53:01.825172 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.812025    4473 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:53:01.825172 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.813657    4473 fs.go:135] Filesystem UUIDs: map[029e9b58-7870-4ae1-a993-6b376facb40f:/dev/nvme0n1p3 351ae0e9-eae6-4b38-868b-f7742a75030b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 21:53:01.825328 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.813686    4473 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/087d0377ddb4aa2adfc070dddbd131183fe85e9449af195f4b7874b6b96767f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/087d0377ddb4aa2adfc070dddbd131183fe85e9449af195f4b7874b6b96767f2/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/319b5d7d5b2e047e272d6e4639deac2c71c67f1c68a6c672d52f330b0b7bc780/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/319b5d7d5b2e047e272d6e4639deac2c71c67f1c68a6c672d52f330b0b7bc780/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a48f337c08359807dacc297270f3d58bd1ca9b2989bc691756e9434805a5416/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a48f337c08359807dacc297270f3d58bd1ca9b2989bc691756e9434805a5416/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c04ba2f75948bec668a42b2992868bcf84922d629761f0bbe67ca23928332c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c04ba2f75948bec668a42b2992868bcf84922d629761f0bbe67ca23928332c4/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d3fe2f59da01374bc7947e14cfb67ac02039b9d7a0ba6a6036cee13bb2d3b6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d3fe2f59da01374bc7947e14cfb67ac02039b9d7a0ba6a6036cee13bb2d3b6f/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95ba7a02912e118a2301ba12ffb28f2abc918aa5ec297848b8a957cff6235870/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95ba7a02912e118a2301ba12ffb28f2abc918aa5ec297848b8a957cff6235870/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/972bcb06f95cbf6872be72074ef4d8708153b5e6409ccfdd4a945ae8ccd21f88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/972bcb06f95cbf6872be72074ef4d8708153b5e6409ccfdd4a945ae8ccd21f88/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2d7957db3c761ac03f31b0a0e04b1010cc70c3ddb947ec72fee2ded6e981b84/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2d7957db3c761ac03f31b0a0e04b1010cc70c3ddb947ec72fee2ded6e981b84/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a97e2000b390e946e0c1ce5f08cda7bb01b9b865a0cd679501f9abfe06639228/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a97e2000b390e946e0c1ce5f08cda7bb01b9b865a0cd679501f9abfe06639228/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b14a7da0cb0afcf499ddc111e19b0691cc9ccf85365b730a1adb2f47217754c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b14a7da0cb0afcf499ddc111e19b0691cc9ccf85365b730a1adb2f47217754c9/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb3171025f33b95209e823494f690f70835c3e48141ef8a53b01609108e16607/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb3171025f33b95209e823494f690f70835c3e48141ef8a53b01609108e16607/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5d48248be9a0e8dda3d52f5736448c3bb5c10fcc9c17e15ddbeee3a0d19347b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5d48248be9a0e8dda3d52f5736448c3bb5c10fcc9c17e15ddbeee3a0d19347b/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7ba18eb6873828ad9148d7cbc35d8c0d98f033645b440cb10bd1fe3ed780173/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7ba18eb6873828ad9148d7cbc35d8c0d98f033645b440cb10bd1fe3ed780173/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e16268348e4444fd54c6ce3d29ad7ebd08975886f2cfd6fa7097122dd8712a95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e16268348e4444fd54c6ce3d29ad7ebd08975886f2cfd6fa7097122dd8712a95/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fab09ade6f257048793a17ee5835c08a606827a22dc71fb0ed9e264cb4c08112/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fab09ade6f257048793a17ee5835c08a606827a22dc71fb0ed9e264cb4c08112/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fabae2396d46b15dab6233392969f273acac4e110a5489f18714fab2b4888bde/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fabae2396d46b15dab6233392969f273acac4e110a5489f18714fab2b4888bde/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14434183-f8cd-4f6e-aab4-d8516d44ec74/volumes/kubernetes.io~projected/kube-api-access-t4qsd:{mountpoint:/var/lib/kubelet/pods/14434183-f8cd-4f6e-aab4-d8516d44ec74/volumes/kubernetes.io~projected/kube-api-access-t4qsd major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/176bb59e-df7f-4b7a-8335-4c23f633ac1f/volumes/kubernetes.io~projected/kube-api-access-qbmhw:{mountpoint:/var/lib/kubelet/pods/176bb59e-df7f-4b7a-8335-4c23f633ac1f/volumes/kubernetes.io~projected/kube-api-access-qbmhw major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/176bb59e-df7f-4b7a-8335-4c23f633ac1f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/176bb59e-df7f-4b7a-8335-4c23f633ac1f/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18aafbda-138f-40ee-9b40-9e1b76163b98/volumes/kubernetes.io~projected/kube-api-access-h8vbm:{mountpoint:/var/lib/kubelet/pods/18aafbda-138f-40ee-9b40-9e1b76163b98/volumes/kubernetes.io~projected/kube-api-access-h8vbm major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3/volumes/kubernetes.io~projected/kube-api-access-5wqh2:{mountpoint:/var/lib/kubelet/pods/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3/volumes/kubernetes.io~projected/kube-api-access-5wqh2 major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3/volumes/kubernetes.io~secret/signing-key major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cced475-7943-49ad-870b-4b7119d62759/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/4cced475-7943-49ad-870b-4b7119d62759/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53d63a68-43fa-4e92-9a39-541cf8983a89/volumes/kubernetes.io~projected/kube-api-access-pcbt6:{mountpoint:/var/lib/kubelet/pods/53d63a68-43fa-4e92-9a39-541cf8983a89/volumes/kubernetes.io~projected/kube-api-access-pcbt6 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60e220ff-82ff-4d25-b53d-6e98dd4da3fd/volumes/kubernetes.io~projected/kube-api-access-l5zlk:{mountpoint:/var/lib/kubelet/pods/60e220ff-82ff-4d25-b53d-6e98dd4da3fd/volumes/kubernetes.io~projected/kube-api-access-l5zlk major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aea3c42-943b-4532-b21d-64ada9922ed2/volumes/kubernetes.io~projected/kube-api-access-kxs96:{mountpoint:/var/lib/kubelet/pods/6aea3c42-943b-4532-b21d-64ada9922ed2/volumes/kubernetes.io~projected/kube-api-access-kxs96 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78e61e36-d7f2-432e-af30-16a093b57e7f/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/78e61e36-d7f2-432e-af30-16a093b57e7f/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78e61e36-d7f2-432e-af30-16a093b57e7f/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/78e61e36-d7f2-432e-af30-16a093b57e7f/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78e61e36-d7f2-432e-af30-16a093b57e7f/volumes/kubernetes.io~projected/kube-api-access-hz8q9:{mountpoint:/var/lib/kubelet/pods/78e61e36-d7f2-432e-af30-16a093b57e7f/volumes/kubernetes.io~projected/kube-api-access-hz8q9 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89f7427f-84b8-4ee1-a4a6-d901b32851f1/volumes/kubernetes.io~projected/kube-api-access-cnq59:{mountpoint:/var/lib/kubelet/pods/89f7427f-84b8-4ee1-a4a6-d901b32851f1/volumes/kubernetes.io~projected/kube-api-access-cnq59 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8faee8d2-4d27-4709-af52-4b5ef32592f6/volumes/kubernetes.io~projected/kube-api-access-m744c:{mountpoint:/var/lib/kubelet/pods/8faee8d2-4d27-4709-af52-4b5ef32592f6/volumes/kubernetes.io~projected/kube-api-access-m744c major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cc773b9-a6f5-48bd-9beb-11efeaae63d9/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/9cc773b9-a6f5-48bd-9beb-11efeaae63d9/volumes/kubernetes.io~secret/agent-certs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4d48390-828e-4c3e-91e8-44264a171fd3/volumes/kubernetes.io~projected/kube-api-access-4lfjn:{mountpoint:/var/lib/kubelet/pods/b4d48390-828e-4c3e-91e8-44264a171fd3/volumes/kubernetes.io~projected/kube-api-access-4lfjn major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d46f22d3-4503-41af-98d9-c641d343de28/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d46f22d3-4503-41af-98d9-c641d343de28/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d46f22d3-4503-41af-98d9-c641d343de28/volumes/kubernetes.io~projected/kube-api-access-kb4lx:{mountpoint:/var/lib/kubelet/pods/d46f22d3-4503-41af-98d9-c641d343de28/volumes/kubernetes.io~projected/kube-api-access-kb4lx major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d46f22d3-4503-41af-98d9-c641d343de28/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d46f22d3-4503-41af-98d9-c641d343de28/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6/volumes/kubernetes.io~projected/kube-api-access-q9qcl:{mountpoint:/var/lib/kubelet/pods/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6/volumes/kubernetes.io~projected/kube-api-access-q9qcl major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f37fd220-ad2c-4fd0-9d24-6b9733ce7d28/volumes/kubernetes.io~projected/kube-api-access-49qz4:{mountpoint:/var/lib/kubelet/pods/f37fd220-ad2c-4fd0-9d24-6b9733ce7d28/volumes/kubernetes.io~projected/kube-api-access-49qz4 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098/volumes/kubernetes.io~projected/kube-api-access-vlk69:{mountpoint:/var/lib/kubelet/pods/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098/volumes/kubernetes.io~projected/kube-api-access-vlk69 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fba18036-6a0c-4aa6-a6b3-b9ab35a57004/volumes/kubernetes.io~projected/kube-api-access-rbjbq:{mountpoint:/var/lib/kubelet/pods/fba18036-6a0c-4aa6-a6b3-b9ab35a57004/volumes/kubernetes.io~projected/kube-api-access-rbjbq major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/92204416b2f3d9802bc45696105945bf6a5cd2cafd78bab993efe1021f67d125/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/64e6a39c74183d33d8ba1947b2e587813f8029c7a3f83b82d60523b71a218784/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/fde48b9ffda97202d13206e873c11c6355059d9f629162ed48ffe17c32833b7d/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/9d777af9f2a900a5a7ea86893fb5aa15f1bb318415a20f81abd6d6246b4fb95b/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/bb47f2784b62e0e52137510dca84ae88adced540d1276d8d542584fe60ff79e4/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/50180cd468e366804f45071afcfacdd8db378e106f8083b850f50319029de65c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/25adc0f55f1d2418d2e69868b838220a4ce5c005d2653f26b6e261c2a52cb276/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/10a9232c2f8d6e59408120a73c927c30c72b0163f875e359dd8aa3919792c06d/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4430ca00008ded87eaeebdc8ec7390fef8b61570d8376b14738e90c4bcea01df/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/f76ab517cfef9a05ec75b600f988fa4aff348e1512f2599d6f5ff8e8c559dcb0/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/6857b63520e9099fbf9da48241b676ebca4d0002842843bc131ea3cbcca45317/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/cf144281aaedd17a475f180e35d3ad8ba5ff4f772e30789b45a313a97b44981a/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e2ad6b4c3a93cfb57d64ee95cfb64a2dc4119f78072b30b97f74ce541828102c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/3790247a50cb26ea6d5500cfd26911b292cc653ca2bac01954213773ee5cfdf9/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/eb7f6988b1af938074687a23b54cefe28aa8fc202a8bbc88e67d25cc72648197/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/20c220a774ece2137261218d96eb902a920b02122f520e1a2fd6459b96d363f9/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/dd16dc669f79753d511e7f20a7b46b5907506a188b0e1f01061d90510bfa4f3b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4e74085d8fb7adee4fccc024226c5b81d596b0215e7e31e067d0313f66494b8a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/9b71091ae49396b69712ce5cf2eea45cd4fc4f8dfb4adc058a5e8ad709c55775/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d2fc7838e71e3fa8e975addc6afd099a7071232025ea3bf297d4d595d02f9fc6/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/eedbe0c88be7445fffadac7b8279326b330d654767eaa76b9d39fc52215ffedf/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/fa4c47ca0cdf7c34a7765a4accc4e98e55be2ac2ad921585122e48bb593c8f58/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/dcde5e36e32a95071f9be0b27f59f92e6cc272b5c411309fcb4262dfd1840ce0/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/7273b62978f6a9c16f839e9a13f476818ad6c21b79a81704da38b516db3c89bd/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/23612e48cdc927113420a54ca28426755ae8f833f607fcd37c14c2c04378bb62/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/362fdee44be70329c7e4bc7e0a6ebf166594878177293abc431a0d6a87f76542/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/bbd2a93bf7e5014954119aa887ce6058dbf45667fcd6005ecda8efd46adae970/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/9a30dfdda96c0ee1ddca097a5100392cd451e6fe16b84e2c53a09e468226f3e2/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/a1f34d987d40ee782370be456c005496fbb8478bd0e203b3cdac4dfe175eccfe/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/40cd0350058cd02e006eced46b11f57b5523123c1c081d90d47355778daab9fa/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/9d7c8438eba8c0a5aeb4364170d452d1156061759e89849f4af7c3c337a21ea3/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/a1d21e19e9a5e2e92b27f3d3bc7cbed44cdce5bdddb5023150d8da8bbcbff6f0/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/161da1c51ead3f759518f95c484ca9ae93a2ca638502480d9dbc1d31e85a4b25/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/69def8880c4ce1f9ca70e327710149022bf5e2ee6bbc8625a04f0ae72e793320/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a175bae1d1147dd39745b3ec53201333931b14d2633afffe538cf315cd83b79e/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/38a1c61648c1363e1c1b6e374f414c4464c985f7924c82e9e22beb1e6bfbaebe/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/c3a13a9bc10f97ea1e7330a9d02bcd56e4c9094e87e64bdb043e4e2117932654/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/a4769ce081e8e058a628ccfd9e510c9ccb110136564d6cf531e931ab07972faa/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/256913d1734a89c624eebe376db3c9e18ece6ced2fd288e300f039e6b682ea2b/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/2924657ac079705aa9fab1b75c0e9b74ad740c329a4dab43e7c8f163d78271ed/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c5a14a139c2a54b5dd4ec2e9f96b28c673230e1914e2080cb267c05fc4e714ad/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.821389    4473 manager.go:217] Machine: {Timestamp:2026-10-30 21:53:01.820913159 +0000 UTC m=+0.057122495 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3135001 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c85a065da967d5d042e836395660c SystemUUID:ec2c85a0-65da-967d-5d04-2e836395660c BootID:a9417424-cc8a-4cca-9448-52839719960a Filesystems:[{Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cc773b9-a6f5-48bd-9beb-11efeaae63d9/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4cced475-7943-49ad-870b-4b7119d62759/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f37fd220-ad2c-4fd0-9d24-6b9733ce7d28/volumes/kubernetes.io~projected/kube-api-access-49qz4 DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7ba18eb6873828ad9148d7cbc35d8c0d98f033645b440cb10bd1fe3ed780173/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:235 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2d7957db3c761ac03f31b0a0e04b1010cc70c3ddb947ec72fee2ded6e981b84/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/176bb59e-df7f-4b7a-8335-4c23f633ac1f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fab09ade6f257048793a17ee5835c08a606827a22dc71fb0ed9e264cb4c08112/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/78e61e36-d7f2-432e-af30-16a093b57e7f/volumes/kubernetes.io~projected/kube-api-access-hz8q9 DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/319b5d7d5b2e047e272d6e4639deac2c71c67f1c68a6c672d52f330b0b7bc780/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d3fe2f59da01374bc7947e14cfb67ac02039b9d7a0ba6a6036cee13bb2d3b6f/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb3171025f33b95209e823494f690f70835c3e48141ef8a53b01609108e16607/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/78e61e36-d7f2-432e-af30-16a093b57e7f/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a97e2000b390e946e0c1ce5f08cda7bb01b9b865a0cd679501f9abfe06639228/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fabae2396d46b15dab6233392969f273acac4e110a5489f18714fab2b4888bde/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/176bb59e-df7f-4b7a-8335-4c23f633ac1f/volumes/kubernetes.io~projected/kube-api-access-qbmhw DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b14a7da0cb0afcf499ddc111e19b0691cc9ccf85365b730a1adb2f47217754c9/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3/volumes/kubernetes.io~projected/kube-api-access-5wqh2 DeviceMajor:0 DeviceMinor:240 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78e61e36-d7f2-432e-af30-16a093b57e7f/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/89f7427f-84b8-4ee1-a4a6-d901b32851f1/volumes/kubernetes.io~projected/kube-api-access-cnq59 DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/972bcb06f95cbf6872be72074ef4d8708153b5e6409ccfdd4a945ae8ccd21f88/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e16268348e4444fd54c6ce3d29ad7ebd08975886f2cfd6fa7097122dd8712a95/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4d48390-828e-4c3e-91e8-44264a171fd3/volumes/kubernetes.io~projected/kube-api-access-4lfjn DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d46f22d3-4503-41af-98d9-c641d343de28/volumes/kubernetes.io~projected/kube-api-access-kb4lx DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d46f22d3-4503-41af-98d9-c641d343de28/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/087d0377ddb4aa2adfc070dddbd131183fe85e9449af195f4b7874b6b96767f2/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6/volumes/kubernetes.io~projected/kube-api-access-q9qcl DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18aafbda-138f-40ee-9b40-9e1b76163b98/volumes/kubernetes.io~projected/kube-api-access-h8vbm DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/53d63a68-43fa-4e92-9a39-541cf8983a89/volumes/kubernetes.io~projected/kube-api-access-pcbt6 DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60e220ff-82ff-4d25-b53d-6e98dd4da3fd/volumes/kubernetes.io~projected/kube-api-access-l5zlk DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8faee8d2-4d27-4709-af52-4b5ef32592f6/volumes/kubernetes.io~projected/kube-api-access-m744c DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5d48248be9a0e8dda3d52f5736448c3bb5c10fcc9c17e15ddbeee3a0d19347b/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/d46f22d3-4503-41af-98d9-c641d343de28/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098/volumes/kubernetes.io~projected/kube-api-access-vlk69 DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6aea3c42-943b-4532-b21d-64ada9922ed2/volumes/kubernetes.io~projected/kube-api-access-kxs96 DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a48f337c08359807dacc297270f3d58bd1ca9b2989bc691756e9434805a5416/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95ba7a02912e118a2301ba12ffb28f2abc918aa5ec297848b8a957cff6235870/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14434183-f8cd-4f6e-aab4-d8516d44ec74/volumes/kubernetes.io~projected/kube-api-access-t4qsd DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fba18036-6a0c-4aa6-a6b3-b9ab35a57004/volumes/kubernetes.io~projected/kube-api-access-rbjbq DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c04ba2f75948bec668a42b2992868bcf84922d629761f0bbe67ca23928332c4/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:087d0377ddb4aa2 MacAddress:3a:b1:96:ed:b7:93 Speed:10000 Mtu:8901} {Name:6d3fe2f59da0137 MacAddress:d6:fe:f0:ee:51:0c Speed:10000 Mtu:8901} {Name:b14a7da0cb0afcf MacAddress:4e:67:31:ed:23:45 Speed:10000 Mtu:8901} {Name:bb3171025f33b95 MacAddress:66:46:23:21:a6:5e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:48:5d:4d:79:95 Speed:0 Mtu:9001} {Name:br-int MacAddress:16:14:54:8e:9f:dd Speed:0 Mtu:8901} {Name:c5d48248be9a0e8 MacAddress:76:42:e5:98:e7:39 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:48:5d:4d:79:95 Speed:0 Mtu:9001} {Name:fabae2396d46b15 MacAddress:4e:9b:73:a9:70:40 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:8a:ef:98:18:aa:f4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fe:12:98:a5:97:af Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.821648    4473 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.821695    4473 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.821995    4473 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.822023    4473 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-162.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.822177    4473 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.822184    4473 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.822198    4473 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.822215    4473 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.822308    4473 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.822468    4473 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.822574    4473 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.822610    4473 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.822623    4473 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.822633    4473 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.822642    4473 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.822650    4473 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.823403    4473 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.823415    4473 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:53:01.825554 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.824073    4473 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:53:01.832487 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.832471    4473 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:53:01.832487 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.832488    4473 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:53:01.832590 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.832581    4473 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 21:53:01.832777 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.832766    4473 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:53:01.832814 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.832795    4473 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:53:01.832888 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.832879    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:53:01.832916 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.832893    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:53:01.832916 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.832900    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:53:01.832916 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.832907    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:53:01.832916 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.832913    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:53:01.833034 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.832920    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:53:01.833034 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.832925    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:53:01.833034 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.832931    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:53:01.833034 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.832938    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:53:01.833034 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.832944    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:53:01.833034 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.832970    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:53:01.833034 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.832981    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:53:01.833034 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.833022    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:53:01.833034 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.833031    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:53:01.836422 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.836409    4473 server.go:1263] "Started kubelet"
Oct 30 21:53:01.836514 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.836464    4473 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:53:01.837170 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.837038    4473 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:53:01.837170 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.837168    4473 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:53:01.837110 ip-10-0-130-162 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:53:01.837922 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.837507    4473 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:53:01.841023 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.840049    4473 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:53:01.843815 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.843583    4473 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:53:01.843815 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.843597    4473 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:53:01.843815 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.843720    4473 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:53:01.843815 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.843734    4473 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:53:01.843815 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.843754    4473 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:53:01.843815 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:01.843786    4473 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:53:01.844620 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.844594    4473 factory.go:55] Registering systemd factory
Oct 30 21:53:01.844688 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.844625    4473 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:53:01.844862 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.844847    4473 factory.go:153] Registering CRI-O factory
Oct 30 21:53:01.844931 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.844867    4473 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:53:01.844931 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.844920    4473 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:53:01.845146 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.844941    4473 factory.go:103] Registering Raw factory
Oct 30 21:53:01.846288 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.845845    4473 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:53:01.846288 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.843827    4473 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:47:23 +0000 UTC" deadline="2028-06-26 19:31:37.510988152 +0000 UTC"
Oct 30 21:53:01.846288 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.845895    4473 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14517h38m35.665098635s"
Oct 30 21:53:01.846288 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846191    4473 manager.go:319] Starting recovery of all containers
Oct 30 21:53:01.846592 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846564    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aea3c42-943b-4532-b21d-64ada9922ed2" volumeName="kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 21:53:01.846625 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846595    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aa2d0cb-19e6-4d3a-8aae-7760c255bef3" volumeName="kubernetes.io/secret/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-signing-key" seLinuxMountContext=""
Oct 30 21:53:01.846625 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846610    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aea3c42-943b-4532-b21d-64ada9922ed2" volumeName="kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:53:01.846625 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846619    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8faee8d2-4d27-4709-af52-4b5ef32592f6" volumeName="kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-service-ca-bundle" seLinuxMountContext=""
Oct 30 21:53:01.846703 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846627    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d46f22d3-4503-41af-98d9-c641d343de28" volumeName="kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-ovnkube-config" seLinuxMountContext=""
Oct 30 21:53:01.846703 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846634    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f37fd220-ad2c-4fd0-9d24-6b9733ce7d28" volumeName="kubernetes.io/projected/f37fd220-ad2c-4fd0-9d24-6b9733ce7d28-kube-api-access-49qz4" seLinuxMountContext=""
Oct 30 21:53:01.846703 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846642    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18aafbda-138f-40ee-9b40-9e1b76163b98" volumeName="kubernetes.io/projected/18aafbda-138f-40ee-9b40-9e1b76163b98-kube-api-access-h8vbm" seLinuxMountContext=""
Oct 30 21:53:01.846703 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846648    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89f7427f-84b8-4ee1-a4a6-d901b32851f1" volumeName="kubernetes.io/configmap/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-daemon-config" seLinuxMountContext=""
Oct 30 21:53:01.846703 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846658    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8faee8d2-4d27-4709-af52-4b5ef32592f6" volumeName="kubernetes.io/projected/8faee8d2-4d27-4709-af52-4b5ef32592f6-kube-api-access-m744c" seLinuxMountContext=""
Oct 30 21:53:01.846703 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846665    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4d48390-828e-4c3e-91e8-44264a171fd3" volumeName="kubernetes.io/empty-dir/b4d48390-828e-4c3e-91e8-44264a171fd3-data-volume" seLinuxMountContext=""
Oct 30 21:53:01.846703 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846673    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4d48390-828e-4c3e-91e8-44264a171fd3" volumeName="kubernetes.io/projected/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-api-access-4lfjn" seLinuxMountContext=""
Oct 30 21:53:01.846703 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846686    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d46f22d3-4503-41af-98d9-c641d343de28" volumeName="kubernetes.io/secret/d46f22d3-4503-41af-98d9-c641d343de28-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 21:53:01.846703 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846694    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbf75704-af7b-44f6-bfcb-b2b2e943f5e6" volumeName="kubernetes.io/projected/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-kube-api-access-q9qcl" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846702    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aa2d0cb-19e6-4d3a-8aae-7760c255bef3" volumeName="kubernetes.io/projected/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-kube-api-access-5wqh2" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846732    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cced475-7943-49ad-870b-4b7119d62759" volumeName="kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846742    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aea3c42-943b-4532-b21d-64ada9922ed2" volumeName="kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846756    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78e61e36-d7f2-432e-af30-16a093b57e7f" volumeName="kubernetes.io/empty-dir/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-tuned" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846769    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78e61e36-d7f2-432e-af30-16a093b57e7f" volumeName="kubernetes.io/projected/78e61e36-d7f2-432e-af30-16a093b57e7f-kube-api-access-hz8q9" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846777    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89f7427f-84b8-4ee1-a4a6-d901b32851f1" volumeName="kubernetes.io/configmap/89f7427f-84b8-4ee1-a4a6-d901b32851f1-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846785    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d46f22d3-4503-41af-98d9-c641d343de28" volumeName="kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-env-overrides" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846792    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d46f22d3-4503-41af-98d9-c641d343de28" volumeName="kubernetes.io/projected/d46f22d3-4503-41af-98d9-c641d343de28-kube-api-access-kb4lx" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846799    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="176bb59e-df7f-4b7a-8335-4c23f633ac1f" volumeName="kubernetes.io/projected/176bb59e-df7f-4b7a-8335-4c23f633ac1f-kube-api-access-qbmhw" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846806    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="176bb59e-df7f-4b7a-8335-4c23f633ac1f" volumeName="kubernetes.io/secret/176bb59e-df7f-4b7a-8335-4c23f633ac1f-cert" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846813    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aa2d0cb-19e6-4d3a-8aae-7760c255bef3" volumeName="kubernetes.io/configmap/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-signing-cabundle" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846820    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53d63a68-43fa-4e92-9a39-541cf8983a89" volumeName="kubernetes.io/empty-dir/53d63a68-43fa-4e92-9a39-541cf8983a89-tmp-dir" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846827    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cc773b9-a6f5-48bd-9beb-11efeaae63d9" volumeName="kubernetes.io/secret/9cc773b9-a6f5-48bd-9beb-11efeaae63d9-agent-certs" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846837    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14434183-f8cd-4f6e-aab4-d8516d44ec74" volumeName="kubernetes.io/projected/14434183-f8cd-4f6e-aab4-d8516d44ec74-kube-api-access-t4qsd" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846848    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89f7427f-84b8-4ee1-a4a6-d901b32851f1" volumeName="kubernetes.io/projected/89f7427f-84b8-4ee1-a4a6-d901b32851f1-kube-api-access-cnq59" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846857    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098" volumeName="kubernetes.io/configmap/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-serviceca" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846865    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba18036-6a0c-4aa6-a6b3-b9ab35a57004" volumeName="kubernetes.io/projected/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-kube-api-access-rbjbq" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846874    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e220ff-82ff-4d25-b53d-6e98dd4da3fd" volumeName="kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846882    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aea3c42-943b-4532-b21d-64ada9922ed2" volumeName="kubernetes.io/projected/6aea3c42-943b-4532-b21d-64ada9922ed2-kube-api-access-kxs96" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846904    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8faee8d2-4d27-4709-af52-4b5ef32592f6" volumeName="kubernetes.io/empty-dir/8faee8d2-4d27-4709-af52-4b5ef32592f6-archives-path" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846914    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cc773b9-a6f5-48bd-9beb-11efeaae63d9" volumeName="kubernetes.io/configmap/9cc773b9-a6f5-48bd-9beb-11efeaae63d9-konnectivity-ca" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846925    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53d63a68-43fa-4e92-9a39-541cf8983a89" volumeName="kubernetes.io/projected/53d63a68-43fa-4e92-9a39-541cf8983a89-kube-api-access-pcbt6" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846932    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8faee8d2-4d27-4709-af52-4b5ef32592f6" volumeName="kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:53:01.846938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846940    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d46f22d3-4503-41af-98d9-c641d343de28" volumeName="kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 21:53:01.847666 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846969    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098" volumeName="kubernetes.io/projected/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-kube-api-access-vlk69" seLinuxMountContext=""
Oct 30 21:53:01.847666 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.846982    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78e61e36-d7f2-432e-af30-16a093b57e7f" volumeName="kubernetes.io/empty-dir/78e61e36-d7f2-432e-af30-16a093b57e7f-tmp" seLinuxMountContext=""
Oct 30 21:53:01.847666 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.847010    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14434183-f8cd-4f6e-aab4-d8516d44ec74" volumeName="kubernetes.io/configmap/14434183-f8cd-4f6e-aab4-d8516d44ec74-iptables-alerter-script" seLinuxMountContext=""
Oct 30 21:53:01.847666 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.847019    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18aafbda-138f-40ee-9b40-9e1b76163b98" volumeName="kubernetes.io/configmap/18aafbda-138f-40ee-9b40-9e1b76163b98-config-volume" seLinuxMountContext=""
Oct 30 21:53:01.847666 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.847028    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18aafbda-138f-40ee-9b40-9e1b76163b98" volumeName="kubernetes.io/empty-dir/18aafbda-138f-40ee-9b40-9e1b76163b98-tmp-dir" seLinuxMountContext=""
Oct 30 21:53:01.847666 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.847036    4473 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:53:01.847666 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.847043    4473 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:53:01.850935 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.850913    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 21:53:01.855001 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.854980    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 21:53:01.879328 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.879288    4473 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:53:01.880623 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.880596    4473 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:53:01.880623 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.880621    4473 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:53:01.880791 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.880639    4473 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:53:01.880791 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:01.880667    4473 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:53:01.895231 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.895203    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 21:53:01.895596 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.895545    4473 generic.go:356] "Generic (PLEG): container finished" podID="d46f22d3-4503-41af-98d9-c641d343de28" containerID="d663386d48bee40654cd125a6da85b38dd8605fb434953a9ab8a4a084ffafd5c" exitCode=1
Oct 30 21:53:01.900686 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.900632    4473 generic.go:356] "Generic (PLEG): container finished" podID="6aea3c42-943b-4532-b21d-64ada9922ed2" containerID="a42cfd90b73e7c8f9a0b9d664c9382a079ff4ce0ff85dd2348601dadc8d1f3fe" exitCode=0
Oct 30 21:53:01.900686 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.900654    4473 generic.go:356] "Generic (PLEG): container finished" podID="6aea3c42-943b-4532-b21d-64ada9922ed2" containerID="0aedaaa6edce2f98ba97c6c453e9f1f9c77c6f6e6eabdaca3296d5acc2f2bca2" exitCode=0
Oct 30 21:53:01.900686 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.900666    4473 generic.go:356] "Generic (PLEG): container finished" podID="6aea3c42-943b-4532-b21d-64ada9922ed2" containerID="4a772e2b9c1ff51b251ca31fd951313b698044e084d73612cdf50cd959a428a7" exitCode=0
Oct 30 21:53:01.900686 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.900674    4473 generic.go:356] "Generic (PLEG): container finished" podID="6aea3c42-943b-4532-b21d-64ada9922ed2" containerID="acc518ef55e5bb300989ec7be340d37b706f99f6382ab39c01efb7116cdf308d" exitCode=0
Oct 30 21:53:01.900686 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.900682    4473 generic.go:356] "Generic (PLEG): container finished" podID="6aea3c42-943b-4532-b21d-64ada9922ed2" containerID="4185b187c2416617b38e6600f8415d1c11ae696faf527f288ed32f74aac8bbf4" exitCode=0
Oct 30 21:53:01.900686 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.900688    4473 generic.go:356] "Generic (PLEG): container finished" podID="6aea3c42-943b-4532-b21d-64ada9922ed2" containerID="95bec18c53211db4775e411555b05cbfe9635dc700a4aded3d0f25d7ca88c4d0" exitCode=0
Oct 30 21:53:01.919792 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.919766    4473 generic.go:356] "Generic (PLEG): container finished" podID="0a78e001a5670be0097c09eab27b6eea" containerID="b3ff85605e6aca3f01e12ca75d477dcc0aa99ab199f9cbf5d6f482be3d5bbccd" exitCode=0
Oct 30 21:53:01.920212 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.920198    4473 manager.go:324] Recovery completed
Oct 30 21:53:01.933614 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.933596    4473 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:53:01.935051 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.935036    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:53:01.935120 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.935065    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:53:01.935120 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.935076    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:53:01.936484 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.936469    4473 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:53:01.936484 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.936482    4473 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:53:01.936593 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.936497    4473 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:53:01.936593 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.936474    4473 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:53:01.936652 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.936616    4473 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 21:53:01.936652 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.936627    4473 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 21:53:01.936652 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.936643    4473 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 21:53:01.936652 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.936648    4473 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 21:53:01.936652 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.936654    4473 policy_none.go:50] "Start"
Oct 30 21:53:01.936785 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.936670    4473 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:53:01.936785 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.936679    4473 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:53:01.936839 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.936801    4473 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 21:53:01.936839 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.936807    4473 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 21:53:01.936839 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.936813    4473 policy_none.go:44] "Start"
Oct 30 21:53:01.937608 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.937576    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:53:01.937705 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.937616    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:53:01.937705 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.937636    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:53:01.940812 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.940788    4473 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:53:01.943928 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:01.943911    4473 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:53:01.944020 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.944008    4473 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:53:01.944057 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:01.944032    4473 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:53:01.944057 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.944041    4473 server.go:87] "Starting device plugin registration server"
Oct 30 21:53:01.944282 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.944272    4473 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:53:01.944313 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.944283    4473 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:53:01.944404 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.944389    4473 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:53:01.944510 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.944479    4473 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:53:01.944569 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.944545    4473 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:53:01.944569 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.944552    4473 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:53:01.945823 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.944599    4473 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:53:01.944508642Z","UUID":"9e622855-054b-4357-9ee3-cc7c1e8bfd90","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:53:01.946002 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.945986    4473 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:53:01.946072 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.946012    4473 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:53:01.946106 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:01.945987    4473 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:53:01.946145 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:01.946109    4473 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-162.us-west-1.compute.internal\" not found"
Oct 30 21:53:01.981595 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.981536    4473 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"]
Oct 30 21:53:01.981747 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.981691    4473 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:53:01.982736 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.982716    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:53:01.982851 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.982744    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:53:01.982851 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.982766    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:53:01.983041 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.983031    4473 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:53:01.983184 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.983154    4473 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:53:01.983927 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.983900    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:53:01.983927 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.983925    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:53:01.984070 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.983935    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:53:01.984070 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.984002    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:53:01.984070 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.984028    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:53:01.984070 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.984038    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:53:01.984189 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:01.984165    4473 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found" node="ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:01.984257 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.984230    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal" event={"ID":"a49dc1ff1f4db21ad91c65b8f36133e5","Type":"ContainerStarted","Data":"ea5d13d1a468b967b7b150db725bb8829122c9304dddf7d920a58a88d616f435"}
Oct 30 21:53:01.984289 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.984264    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal" event={"ID":"a49dc1ff1f4db21ad91c65b8f36133e5","Type":"ContainerStarted","Data":"fab09ade6f257048793a17ee5835c08a606827a22dc71fb0ed9e264cb4c08112"}
Oct 30 21:53:01.984289 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.984277    4473 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:53:01.984289 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.984285    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal" event={"ID":"0a78e001a5670be0097c09eab27b6eea","Type":"ContainerStarted","Data":"8a2ba6703ff5bc2739f8d5b2f08b18b481be78e494a0d181211112f4b17728f0"}
Oct 30 21:53:01.984380 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.984298    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal" event={"ID":"0a78e001a5670be0097c09eab27b6eea","Type":"ContainerDied","Data":"b3ff85605e6aca3f01e12ca75d477dcc0aa99ab199f9cbf5d6f482be3d5bbccd"}
Oct 30 21:53:01.984380 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.984310    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal" event={"ID":"0a78e001a5670be0097c09eab27b6eea","Type":"ContainerStarted","Data":"d7ba18eb6873828ad9148d7cbc35d8c0d98f033645b440cb10bd1fe3ed780173"}
Oct 30 21:53:01.985177 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.985164    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:53:01.985248 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.985188    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:53:01.985248 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:01.985197    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:53:01.985348 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:01.985336    4473 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-162.us-west-1.compute.internal\" not found" node="ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.032211 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.032185    4473 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:53:02.033199 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.033181    4473 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:53:02.033262 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.033248    4473 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 21:53:02.033629 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.033614    4473 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:53:02.044638 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.044613    4473 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.044711 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.044639    4473 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:53:02.045846 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.045830    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:53:02.045981 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.045862    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:53:02.045981 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.045877    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:53:02.045981 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.045911    4473 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.084295 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.084266    4473 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:53:02.124580 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.124555    4473 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:53:02.124684 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:02.124603    4473 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.124684 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.124616    4473 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.133769 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.133738    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a78e001a5670be0097c09eab27b6eea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"0a78e001a5670be0097c09eab27b6eea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.133860 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.133770    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a78e001a5670be0097c09eab27b6eea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"0a78e001a5670be0097c09eab27b6eea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.133860 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.133790    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a49dc1ff1f4db21ad91c65b8f36133e5-config\") pod \"kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"a49dc1ff1f4db21ad91c65b8f36133e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.177086 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.177019    4473 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.177187 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.177092    4473 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.177649 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.177620    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeNotReady"
Oct 30 21:53:02.177749 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.177661    4473 setters.go:546] "Node became not ready" node="ip-10-0-130-162.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T21:53:02Z","lastTransitionTime":"2026-10-30T21:53:02Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 21:53:02.185121 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.185100    4473 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:53:02.185195 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:02.185142    4473 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.199748 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.199729    4473 nodeinfomanager.go:471] existing CSINode "ip-10-0-130-162.us-west-1.compute.internal" is owned by different node (oldNodeID="8de89da3-b335-4083-a90b-bf08c968b622", newNodeID=""), cleaning up...
Oct 30 21:53:02.234111 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.234092    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a49dc1ff1f4db21ad91c65b8f36133e5-config\") pod \"kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"a49dc1ff1f4db21ad91c65b8f36133e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.234189 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.234115    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a78e001a5670be0097c09eab27b6eea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"0a78e001a5670be0097c09eab27b6eea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.234189 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.234132    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a78e001a5670be0097c09eab27b6eea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"0a78e001a5670be0097c09eab27b6eea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.234189 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.234175    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a78e001a5670be0097c09eab27b6eea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"0a78e001a5670be0097c09eab27b6eea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.234275 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.234205    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a49dc1ff1f4db21ad91c65b8f36133e5-config\") pod \"kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"a49dc1ff1f4db21ad91c65b8f36133e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.234275 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.234209    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a78e001a5670be0097c09eab27b6eea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal\" (UID: \"0a78e001a5670be0097c09eab27b6eea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.449626 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.449552    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-162.us-west-1.compute.internal" event="NodeReady"
Oct 30 21:53:02.449626 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.449628    4473 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:53:02.823466 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.823392    4473 apiserver.go:52] "Watching apiserver"
Oct 30 21:53:02.922398 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.922369    4473 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.922581 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.922522    4473 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:02.981433 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.981403    4473 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:53:02.981861 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.981841    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-28mbh","openshift-ovn-kubernetes/ovnkube-node-l8xr5","openshift-service-ca/service-ca-5fffc97bdf-72s7t","openshift-dns/node-resolver-x2gfj","openshift-insights/periodic-gathering-w68zw-mzlkv","openshift-multus/network-metrics-daemon-kprst","openshift-network-diagnostics/network-check-target-b5h24","openshift-insights/insights-runtime-extractor-nf7t4","kube-system/konnectivity-agent-2mskp","openshift-image-registry/node-ca-pcjh8","openshift-ingress-canary/ingress-canary-8w5d6","openshift-multus/multus-lh5dm","openshift-cluster-node-tuning-operator/tuned-7hbsq","kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-j68lz","openshift-dns/dns-default-wx8d9","openshift-kube-storage-version-migrator/migrator-66df5d496b-dlnp8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb","kube-system/global-pull-secret-syncer-4tlxp"]
Oct 30 21:53:02.985410 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.985358    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:53:02.988877 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.988857    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:53:02.989331 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.989315    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:02.991511 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.991495    4473 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:53:02.995893 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:02.995881    4473 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:53:03.032485 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:03.032452    4473 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:03.032485 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:03.032457    4473 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal"
Oct 30 21:53:03.035022 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.035006    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:53:03.035417 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.035402    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:53:03.035542 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.035529    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gmfbd\""
Oct 30 21:53:03.035614 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.035597    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vk8r7\""
Oct 30 21:53:03.035787 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.035773    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:53:03.035821 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.035777    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:53:03.045400 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.045387    4473 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:53:03.114287 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114218    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:53:03.114287 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114252    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:53:03.114287 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114287    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:53:03.114484 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114314    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:53:03.114484 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114327    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:53:03.114484 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114393    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:53:03.114484 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114418    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:53:03.114484 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114483    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:53:03.114656 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114552    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:53:03.114656 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114605    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:53:03.114656 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114636    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:53:03.114768 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114731    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:53:03.114768 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114747    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:53:03.114871 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114774    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:53:03.114871 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114785    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:53:03.114871 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114776    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:53:03.114871 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114804    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:53:03.114871 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114817    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:53:03.115065 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114892    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 21:53:03.115065 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114896    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 21:53:03.115065 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114891    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 21:53:03.115065 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114986    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:53:03.115065 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.114991    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:53:03.115065 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.115033    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:53:03.115225 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.115148    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:53:03.115225 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.115181    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:53:03.115225 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.115208    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:53:03.115225 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.115223    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:53:03.115359 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.115224    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:53:03.115359 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.115252    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:53:03.115436 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.115360    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:53:03.115436 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.115377    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:53:03.115436 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.115393    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:53:03.115693 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.115681    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:53:03.139109 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139092    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89f7427f-84b8-4ee1-a4a6-d901b32851f1-cni-binary-copy\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.139152 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139117    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-ovnkube-script-lib\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.139152 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139135    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-host\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.139230 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139155    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.139230 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139209    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-k8s-cni-cncf-io\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.139320 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139241    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-kubelet\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.139320 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139287    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlk69\" (UniqueName: \"kubernetes.io/projected/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-kube-api-access-vlk69\") pod \"node-ca-pcjh8\" (UID: \"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098\") " pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:53:03.139460 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139363    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wqh2\" (UniqueName: \"kubernetes.io/projected/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-kube-api-access-5wqh2\") pod \"service-ca-5fffc97bdf-72s7t\" (UID: \"2aa2d0cb-19e6-4d3a-8aae-7760c255bef3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t"
Oct 30 21:53:03.139460 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139428    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-ovnkube-script-lib\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.139460 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139442    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9qcl\" (UniqueName: \"kubernetes.io/projected/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-kube-api-access-q9qcl\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:53:03.139589 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139485    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-node-log\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.139589 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139521    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-cni-bin\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.139589 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139544    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-var-lib-kubelet\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.139589 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139577    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcbt6\" (UniqueName: \"kubernetes.io/projected/53d63a68-43fa-4e92-9a39-541cf8983a89-kube-api-access-pcbt6\") pod \"node-resolver-x2gfj\" (UID: \"53d63a68-43fa-4e92-9a39-541cf8983a89\") " pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:53:03.139757 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139626    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-registration-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.139757 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139650    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89f7427f-84b8-4ee1-a4a6-d901b32851f1-cni-binary-copy\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.139757 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139700    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14434183-f8cd-4f6e-aab4-d8516d44ec74-iptables-alerter-script\") pod \"iptables-alerter-j68lz\" (UID: \"14434183-f8cd-4f6e-aab4-d8516d44ec74\") " pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:53:03.139837 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139769    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-systemd-units\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.139837 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139785    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.139837 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139812    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/18aafbda-138f-40ee-9b40-9e1b76163b98-config-volume\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:53:03.139923 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139841    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18aafbda-138f-40ee-9b40-9e1b76163b98-tmp-dir\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:53:03.139923 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139857    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-modprobe-d\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.139923 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139879    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysctl-d\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.140065 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139933    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18aafbda-138f-40ee-9b40-9e1b76163b98-tmp-dir\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:53:03.140065 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.139974    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14434183-f8cd-4f6e-aab4-d8516d44ec74-iptables-alerter-script\") pod \"iptables-alerter-j68lz\" (UID: \"14434183-f8cd-4f6e-aab4-d8516d44ec74\") " pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:53:03.140065 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140011    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-system-cni-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.140065 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140047    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5zlk\" (UniqueName: \"kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk\") pod \"network-check-target-b5h24\" (UID: \"60e220ff-82ff-4d25-b53d-6e98dd4da3fd\") " pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:53:03.140192 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140109    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/18aafbda-138f-40ee-9b40-9e1b76163b98-config-volume\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:53:03.140192 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140155    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-run-netns\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.140192 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140185    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-signing-key\") pod \"service-ca-5fffc97bdf-72s7t\" (UID: \"2aa2d0cb-19e6-4d3a-8aae-7760c255bef3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t"
Oct 30 21:53:03.140299 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140209    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-cnibin\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:53:03.140299 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140245    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-ovn\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.140299 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140263    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-log-socket\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.140299 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140278    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-run-ovn-kubernetes\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.140299 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140293    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4d48390-828e-4c3e-91e8-44264a171fd3-crio-socket\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:03.140504 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140317    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-signing-cabundle\") pod \"service-ca-5fffc97bdf-72s7t\" (UID: \"2aa2d0cb-19e6-4d3a-8aae-7760c255bef3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t"
Oct 30 21:53:03.140504 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140352    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:53:03.140504 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140468    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53d63a68-43fa-4e92-9a39-541cf8983a89-hosts-file\") pod \"node-resolver-x2gfj\" (UID: \"53d63a68-43fa-4e92-9a39-541cf8983a89\") " pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:53:03.140504 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140498    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:03.140663 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140514    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-cni-bin\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.140663 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140554    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-multus-certs\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.140663 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140558    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-signing-cabundle\") pod \"service-ca-5fffc97bdf-72s7t\" (UID: \"2aa2d0cb-19e6-4d3a-8aae-7760c255bef3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t"
Oct 30 21:53:03.140663 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140574    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-trusted-ca-bundle\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:53:03.140663 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140596    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-etc-selinux\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.140663 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140625    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:53:03.140663 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140649    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:53:03.141013 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140692    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxs96\" (UniqueName: \"kubernetes.io/projected/6aea3c42-943b-4532-b21d-64ada9922ed2-kube-api-access-kxs96\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:53:03.141013 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140725    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-etc-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.141013 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140751    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:53:03.141013 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140777    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8vbm\" (UniqueName: \"kubernetes.io/projected/18aafbda-138f-40ee-9b40-9e1b76163b98-kube-api-access-h8vbm\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:53:03.141013 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140801    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysctl-conf\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.141013 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140825    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:53:03.141013 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140847    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-socket-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.141013 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140935    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-service-ca-bundle\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:53:03.141013 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.140985    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-kubelet\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.141320 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141057    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-lib-modules\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.141320 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141086    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-service-ca-bundle\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:53:03.141320 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141089    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-sys-fs\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.141320 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141127    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4qsd\" (UniqueName: \"kubernetes.io/projected/14434183-f8cd-4f6e-aab4-d8516d44ec74-kube-api-access-t4qsd\") pod \"iptables-alerter-j68lz\" (UID: \"14434183-f8cd-4f6e-aab4-d8516d44ec74\") " pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:53:03.141320 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141152    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-cnibin\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.141320 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141175    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-conf-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.141320 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141207    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/176bb59e-df7f-4b7a-8335-4c23f633ac1f-cert\") pod \"ingress-canary-8w5d6\" (UID: \"176bb59e-df7f-4b7a-8335-4c23f633ac1f\") " pod="openshift-ingress-canary/ingress-canary-8w5d6"
Oct 30 21:53:03.141320 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141285    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-cni-binary-copy\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:53:03.141613 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141337    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-etc-kubernetes\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.141613 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141354    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-kubernetes\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.141613 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141371    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-tuned\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.141613 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141387    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78e61e36-d7f2-432e-af30-16a093b57e7f-tmp\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.141613 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141402    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-daemon-config\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.141613 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141459    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysconfig\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.141613 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141482    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-cni-binary-copy\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:53:03.141613 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141498    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-sys\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.141613 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141522    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78e61e36-d7f2-432e-af30-16a093b57e7f-tmp\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.141613 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141538    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-tuned\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.141613 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141613    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-daemon-config\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.141933 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141918    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz8q9\" (UniqueName: \"kubernetes.io/projected/78e61e36-d7f2-432e-af30-16a093b57e7f-kube-api-access-hz8q9\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.141992 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141942    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lfjn\" (UniqueName: \"kubernetes.io/projected/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-api-access-4lfjn\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:03.141992 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141975    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-cni-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.142048 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.141995    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-cni-multus\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.142048 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142018    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4cced475-7943-49ad-870b-4b7119d62759-dbus\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:53:03.142048 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142034    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8faee8d2-4d27-4709-af52-4b5ef32592f6-archives-path\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:53:03.142126 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142052    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-cni-netd\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.142126 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142096    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-device-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.142126 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142115    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:03.142205 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142133    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-hostroot\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.142205 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142133    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8faee8d2-4d27-4709-af52-4b5ef32592f6-archives-path\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:53:03.142205 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142165    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-serviceca\") pod \"node-ca-pcjh8\" (UID: \"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098\") " pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:53:03.142205 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142188    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9cc773b9-a6f5-48bd-9beb-11efeaae63d9-agent-certs\") pod \"konnectivity-agent-2mskp\" (UID: \"9cc773b9-a6f5-48bd-9beb-11efeaae63d9\") " pod="kube-system/konnectivity-agent-2mskp"
Oct 30 21:53:03.142205 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142203    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9cc773b9-a6f5-48bd-9beb-11efeaae63d9-konnectivity-ca\") pod \"konnectivity-agent-2mskp\" (UID: \"9cc773b9-a6f5-48bd-9beb-11efeaae63d9\") " pod="kube-system/konnectivity-agent-2mskp"
Oct 30 21:53:03.142343 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142218    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb4lx\" (UniqueName: \"kubernetes.io/projected/d46f22d3-4503-41af-98d9-c641d343de28-kube-api-access-kb4lx\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.142343 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142285    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-netns\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.142343 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142316    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnq59\" (UniqueName: \"kubernetes.io/projected/89f7427f-84b8-4ee1-a4a6-d901b32851f1-kube-api-access-cnq59\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.142429 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142345    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4cced475-7943-49ad-870b-4b7119d62759-kubelet-config\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:53:03.142429 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142408    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:53:03.142500 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142438    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.142500 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142467    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbjbq\" (UniqueName: \"kubernetes.io/projected/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-kube-api-access-rbjbq\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.142500 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142476    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9cc773b9-a6f5-48bd-9beb-11efeaae63d9-konnectivity-ca\") pod \"konnectivity-agent-2mskp\" (UID: \"9cc773b9-a6f5-48bd-9beb-11efeaae63d9\") " pod="kube-system/konnectivity-agent-2mskp"
Oct 30 21:53:03.142500 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142478    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-serviceca\") pod \"node-ca-pcjh8\" (UID: \"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098\") " pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:53:03.142500 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142494    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14434183-f8cd-4f6e-aab4-d8516d44ec74-host-slash\") pod \"iptables-alerter-j68lz\" (UID: \"14434183-f8cd-4f6e-aab4-d8516d44ec74\") " pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:53:03.142664 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142647    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-socket-dir-parent\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.142698 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142672    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-host\") pod \"node-ca-pcjh8\" (UID: \"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098\") " pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:53:03.142698 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142692    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49qz4\" (UniqueName: \"kubernetes.io/projected/f37fd220-ad2c-4fd0-9d24-6b9733ce7d28-kube-api-access-49qz4\") pod \"migrator-66df5d496b-dlnp8\" (UID: \"f37fd220-ad2c-4fd0-9d24-6b9733ce7d28\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dlnp8"
Oct 30 21:53:03.142751 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142714    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m744c\" (UniqueName: \"kubernetes.io/projected/8faee8d2-4d27-4709-af52-4b5ef32592f6-kube-api-access-m744c\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:53:03.142875 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142862    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-system-cni-dir\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:53:03.142905 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142894    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:53:03.143003 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.142987    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-slash\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.143038 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.143018    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-systemd\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.143066 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.143047    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-env-overrides\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.143108 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.143079    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d46f22d3-4503-41af-98d9-c641d343de28-ovn-node-metrics-cert\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.143138 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.143091    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6aea3c42-943b-4532-b21d-64ada9922ed2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:53:03.143138 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.143129    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbmhw\" (UniqueName: \"kubernetes.io/projected/176bb59e-df7f-4b7a-8335-4c23f633ac1f-kube-api-access-qbmhw\") pod \"ingress-canary-8w5d6\" (UID: \"176bb59e-df7f-4b7a-8335-4c23f633ac1f\") " pod="openshift-ingress-canary/ingress-canary-8w5d6"
Oct 30 21:53:03.143199 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.143164    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-var-lib-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.143199 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.143192    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-env-overrides\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.143251 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.143191    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-ovnkube-config\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.143251 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.143231    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-systemd\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.143306 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.143263    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-run\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.143306 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.143294    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d63a68-43fa-4e92-9a39-541cf8983a89-tmp-dir\") pod \"node-resolver-x2gfj\" (UID: \"53d63a68-43fa-4e92-9a39-541cf8983a89\") " pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:53:03.143357 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.143345    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4d48390-828e-4c3e-91e8-44264a171fd3-data-volume\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:03.143387 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.143372    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d46f22d3-4503-41af-98d9-c641d343de28-ovnkube-config\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.143425 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.143413    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d63a68-43fa-4e92-9a39-541cf8983a89-tmp-dir\") pod \"node-resolver-x2gfj\" (UID: \"53d63a68-43fa-4e92-9a39-541cf8983a89\") " pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:53:03.143456 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.143432    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4d48390-828e-4c3e-91e8-44264a171fd3-data-volume\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:03.163642 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.163609    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:03.165592 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.165550    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:53:03.165592 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.165548    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:53:03.166458 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.165840    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:53:03.166458 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.166022    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mgstz\""
Oct 30 21:53:03.166880 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.166843    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:53:03.166880 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.166860    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:53:03.167071 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.167055    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:53:03.167125 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.167074    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gg6dx\""
Oct 30 21:53:03.167125 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.167104    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fmdw6\""
Oct 30 21:53:03.167344 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.167325    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q4htg\""
Oct 30 21:53:03.168386 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.167728    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-qt896\""
Oct 30 21:53:03.168386 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.167831    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:53:03.168386 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.167850    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h6mh8\""
Oct 30 21:53:03.168386 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.167899    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gsd79\""
Oct 30 21:53:03.168386 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.168036    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jrvhp\""
Oct 30 21:53:03.168386 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.168166    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7bphz\""
Oct 30 21:53:03.168386 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.168263    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:53:03.168386 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.168281    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:53:03.168386 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.168289    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6q7ft\""
Oct 30 21:53:03.168386 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.168343    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hqgxz\""
Oct 30 21:53:03.168844 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.168444    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ppl5h\""
Oct 30 21:53:03.168844 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.168494    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 21:53:03.168844 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.168528    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jfz4q\""
Oct 30 21:53:03.168844 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.168575    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bdqch\""
Oct 30 21:53:03.168844 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.168610    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gqgtq\""
Oct 30 21:53:03.168844 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.168633    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ww8ls\""
Oct 30 21:53:03.170518 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.170503    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-signing-key\") pod \"service-ca-5fffc97bdf-72s7t\" (UID: \"2aa2d0cb-19e6-4d3a-8aae-7760c255bef3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t"
Oct 30 21:53:03.171289 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.171272    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-trusted-ca-bundle\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:53:03.171417 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.171404    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/176bb59e-df7f-4b7a-8335-4c23f633ac1f-cert\") pod \"ingress-canary-8w5d6\" (UID: \"176bb59e-df7f-4b7a-8335-4c23f633ac1f\") " pod="openshift-ingress-canary/ingress-canary-8w5d6"
Oct 30 21:53:03.172882 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.172865    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9cc773b9-a6f5-48bd-9beb-11efeaae63d9-agent-certs\") pod \"konnectivity-agent-2mskp\" (UID: \"9cc773b9-a6f5-48bd-9beb-11efeaae63d9\") " pod="kube-system/konnectivity-agent-2mskp"
Oct 30 21:53:03.172977 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.172945    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cced475-7943-49ad-870b-4b7119d62759-original-pull-secret\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:53:03.173906 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.173891    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d46f22d3-4503-41af-98d9-c641d343de28-ovn-node-metrics-cert\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.195504 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.195478    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlk69\" (UniqueName: \"kubernetes.io/projected/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-kube-api-access-vlk69\") pod \"node-ca-pcjh8\" (UID: \"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098\") " pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:53:03.195589 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.195508    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9qcl\" (UniqueName: \"kubernetes.io/projected/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-kube-api-access-q9qcl\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:53:03.195589 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.195556    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5zlk\" (UniqueName: \"kubernetes.io/projected/60e220ff-82ff-4d25-b53d-6e98dd4da3fd-kube-api-access-l5zlk\") pod \"network-check-target-b5h24\" (UID: \"60e220ff-82ff-4d25-b53d-6e98dd4da3fd\") " pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:53:03.195939 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.195924    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcbt6\" (UniqueName: \"kubernetes.io/projected/53d63a68-43fa-4e92-9a39-541cf8983a89-kube-api-access-pcbt6\") pod \"node-resolver-x2gfj\" (UID: \"53d63a68-43fa-4e92-9a39-541cf8983a89\") " pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:53:03.196333 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.196319    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wqh2\" (UniqueName: \"kubernetes.io/projected/2aa2d0cb-19e6-4d3a-8aae-7760c255bef3-kube-api-access-5wqh2\") pod \"service-ca-5fffc97bdf-72s7t\" (UID: \"2aa2d0cb-19e6-4d3a-8aae-7760c255bef3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t"
Oct 30 21:53:03.198321 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.198307    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4qsd\" (UniqueName: \"kubernetes.io/projected/14434183-f8cd-4f6e-aab4-d8516d44ec74-kube-api-access-t4qsd\") pod \"iptables-alerter-j68lz\" (UID: \"14434183-f8cd-4f6e-aab4-d8516d44ec74\") " pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:53:03.199990 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.199970    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8vbm\" (UniqueName: \"kubernetes.io/projected/18aafbda-138f-40ee-9b40-9e1b76163b98-kube-api-access-h8vbm\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:53:03.200214 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.200193    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxs96\" (UniqueName: \"kubernetes.io/projected/6aea3c42-943b-4532-b21d-64ada9922ed2-kube-api-access-kxs96\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:53:03.201602 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.201583    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lfjn\" (UniqueName: \"kubernetes.io/projected/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-api-access-4lfjn\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:03.201602 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.201597    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnq59\" (UniqueName: \"kubernetes.io/projected/89f7427f-84b8-4ee1-a4a6-d901b32851f1-kube-api-access-cnq59\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.202116 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.202098    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb4lx\" (UniqueName: \"kubernetes.io/projected/d46f22d3-4503-41af-98d9-c641d343de28-kube-api-access-kb4lx\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.202400 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.202385    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m744c\" (UniqueName: \"kubernetes.io/projected/8faee8d2-4d27-4709-af52-4b5ef32592f6-kube-api-access-m744c\") pod \"periodic-gathering-w68zw-mzlkv\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") " pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:53:03.202894 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.202877    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz8q9\" (UniqueName: \"kubernetes.io/projected/78e61e36-d7f2-432e-af30-16a093b57e7f-kube-api-access-hz8q9\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.203254 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.203240    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbjbq\" (UniqueName: \"kubernetes.io/projected/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-kube-api-access-rbjbq\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.203392 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.203374    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49qz4\" (UniqueName: \"kubernetes.io/projected/f37fd220-ad2c-4fd0-9d24-6b9733ce7d28-kube-api-access-49qz4\") pod \"migrator-66df5d496b-dlnp8\" (UID: \"f37fd220-ad2c-4fd0-9d24-6b9733ce7d28\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dlnp8"
Oct 30 21:53:03.203472 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.203456    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbmhw\" (UniqueName: \"kubernetes.io/projected/176bb59e-df7f-4b7a-8335-4c23f633ac1f-kube-api-access-qbmhw\") pod \"ingress-canary-8w5d6\" (UID: \"176bb59e-df7f-4b7a-8335-4c23f633ac1f\") " pod="openshift-ingress-canary/ingress-canary-8w5d6"
Oct 30 21:53:03.243635 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243610    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-var-lib-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.243635 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243636    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-systemd\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.243784 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243652    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-run\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.243784 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243666    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-host\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.243784 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243705    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-var-lib-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.243784 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243724    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-systemd\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.243784 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243766    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-host\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243796    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243803    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-run\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243813    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-k8s-cni-cncf-io\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243830    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-kubelet\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243840    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243866    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-kubelet\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243868    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-node-log\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243844    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-node-log\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243870    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-k8s-cni-cncf-io\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243888    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-cni-bin\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243928    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-var-lib-kubelet\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243962    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-registration-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243978    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-systemd-units\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243928    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-cni-bin\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.243992    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244008    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-modprobe-d\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244024    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-var-lib-kubelet\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.244025 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244031    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysctl-d\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244048    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-systemd-units\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244051    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-registration-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244072    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244102    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-system-cni-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244104    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-modprobe-d\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244124    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-run-netns\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244128    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysctl-d\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244144    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-cnibin\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244161    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-ovn\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244167    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-run-netns\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244178    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-log-socket\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244191    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-cnibin\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244193    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-run-ovn-kubernetes\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244204    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-ovn\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244161    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-system-cni-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244223    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-run-ovn-kubernetes\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244235    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4d48390-828e-4c3e-91e8-44264a171fd3-crio-socket\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244254    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244269    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53d63a68-43fa-4e92-9a39-541cf8983a89-hosts-file\") pod \"node-resolver-x2gfj\" (UID: \"53d63a68-43fa-4e92-9a39-541cf8983a89\") " pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244292    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244306    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-cni-bin\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244312    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4d48390-828e-4c3e-91e8-44264a171fd3-crio-socket\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:03.244348    4473 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244355    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-multus-certs\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244356    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53d63a68-43fa-4e92-9a39-541cf8983a89-hosts-file\") pod \"node-resolver-x2gfj\" (UID: \"53d63a68-43fa-4e92-9a39-541cf8983a89\") " pod="openshift-dns/node-resolver-x2gfj"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244210    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-log-socket\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244320    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-multus-certs\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244384    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-cni-bin\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244400    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-etc-selinux\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:03.244414    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs podName:dbf75704-af7b-44f6-bfcb-b2b2e943f5e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:03.74439864 +0000 UTC m=+1.980607983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs") pod "network-metrics-daemon-kprst" (UID: "dbf75704-af7b-44f6-bfcb-b2b2e943f5e6") : secret "metrics-daemon-secret" not found
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244437    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244445    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-etc-selinux\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244461    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-etc-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244475    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244496    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244508    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysctl-conf\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244534    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-etc-openvswitch\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244538    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-socket-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:03.244565    4473 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244573    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-kubelet\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244587    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-socket-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244596    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-lib-modules\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:03.244616    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls podName:18aafbda-138f-40ee-9b40-9e1b76163b98 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:03.744600728 +0000 UTC m=+1.980810071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls") pod "dns-default-wx8d9" (UID: "18aafbda-138f-40ee-9b40-9e1b76163b98") : secret "dns-default-metrics-tls" not found
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244619    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysctl-conf\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244628    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-kubelet\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244641    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-sys-fs\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244657    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-cnibin\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244671    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-conf-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244668    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-lib-modules\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244695    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-etc-kubernetes\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244711    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-kubernetes\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244713    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-cnibin\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244731    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-conf-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244741    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-etc-kubernetes\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244733    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysconfig\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.244702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244709    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-sys-fs\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244767    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-sys\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244782    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-sysconfig\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244789    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-etc-kubernetes\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244803    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-cni-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244818    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4d48390-828e-4c3e-91e8-44264a171fd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244850    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-cni-multus\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244856    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-cni-dir\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244822    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-var-lib-cni-multus\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244815    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78e61e36-d7f2-432e-af30-16a093b57e7f-sys\") pod \"tuned-7hbsq\" (UID: \"78e61e36-d7f2-432e-af30-16a093b57e7f\") " pod="openshift-cluster-node-tuning-operator/tuned-7hbsq"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244888    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4cced475-7943-49ad-870b-4b7119d62759-dbus\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244913    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-cni-netd\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244929    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-device-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244945    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244979    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-hostroot\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244984    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4cced475-7943-49ad-870b-4b7119d62759-dbus\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244996    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-netns\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245003    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fba18036-6a0c-4aa6-a6b3-b9ab35a57004-device-dir\") pod \"aws-ebs-csi-driver-node-r2rdb\" (UID: \"fba18036-6a0c-4aa6-a6b3-b9ab35a57004\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r2rdb"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245012    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4cced475-7943-49ad-870b-4b7119d62759-kubelet-config\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.244978    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-cni-netd\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245015    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-hostroot\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:03.245037    4473 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245063    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4cced475-7943-49ad-870b-4b7119d62759-kubelet-config\") pod \"global-pull-secret-syncer-4tlxp\" (UID: \"4cced475-7943-49ad-870b-4b7119d62759\") " pod="kube-system/global-pull-secret-syncer-4tlxp"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245071    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245048    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-host-run-netns\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:03.245089    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls podName:b4d48390-828e-4c3e-91e8-44264a171fd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:03.745079274 +0000 UTC m=+1.981288615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nf7t4" (UID: "b4d48390-828e-4c3e-91e8-44264a171fd3") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245108    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245120    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14434183-f8cd-4f6e-aab4-d8516d44ec74-host-slash\") pod \"iptables-alerter-j68lz\" (UID: \"14434183-f8cd-4f6e-aab4-d8516d44ec74\") " pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245136    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-socket-dir-parent\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245151    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-host\") pod \"node-ca-pcjh8\" (UID: \"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098\") " pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245154    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14434183-f8cd-4f6e-aab4-d8516d44ec74-host-slash\") pod \"iptables-alerter-j68lz\" (UID: \"14434183-f8cd-4f6e-aab4-d8516d44ec74\") " pod="openshift-network-operator/iptables-alerter-j68lz"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245167    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-system-cni-dir\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245179    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098-host\") pod \"node-ca-pcjh8\" (UID: \"f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098\") " pod="openshift-image-registry/node-ca-pcjh8"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245183    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-slash\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245195    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89f7427f-84b8-4ee1-a4a6-d901b32851f1-multus-socket-dir-parent\") pod \"multus-lh5dm\" (UID: \"89f7427f-84b8-4ee1-a4a6-d901b32851f1\") " pod="openshift-multus/multus-lh5dm"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245202    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6aea3c42-943b-4532-b21d-64ada9922ed2-system-cni-dir\") pod \"multus-additional-cni-plugins-28mbh\" (UID: \"6aea3c42-943b-4532-b21d-64ada9922ed2\") " pod="openshift-multus/multus-additional-cni-plugins-28mbh"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245197    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-systemd\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245227    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-host-slash\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.246369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.245250    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d46f22d3-4503-41af-98d9-c641d343de28-run-systemd\") pod \"ovnkube-node-l8xr5\" (UID: \"d46f22d3-4503-41af-98d9-c641d343de28\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:03.291935 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.291917    4473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:53:03.748592 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.748558    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:03.748738 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.748630    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:53:03.748738 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:03.748661    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:53:03.748738 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:03.748704    4473 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:53:03.748845 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:03.748747    4473 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:53:03.748845 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:03.748768    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls podName:b4d48390-828e-4c3e-91e8-44264a171fd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:04.748751479 +0000 UTC m=+2.984960803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nf7t4" (UID: "b4d48390-828e-4c3e-91e8-44264a171fd3") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:53:03.748845 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:03.748767    4473 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:53:03.748845 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:03.748789    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls podName:18aafbda-138f-40ee-9b40-9e1b76163b98 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:04.74877913 +0000 UTC m=+2.984988467 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls") pod "dns-default-wx8d9" (UID: "18aafbda-138f-40ee-9b40-9e1b76163b98") : secret "dns-default-metrics-tls" not found
Oct 30 21:53:03.748845 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:03.748826    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs podName:dbf75704-af7b-44f6-bfcb-b2b2e943f5e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:04.748808152 +0000 UTC m=+2.985017493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs") pod "network-metrics-daemon-kprst" (UID: "dbf75704-af7b-44f6-bfcb-b2b2e943f5e6") : secret "metrics-daemon-secret" not found
Oct 30 21:53:04.756244 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:04.756164    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:53:04.756244 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:04.756218    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:53:04.756729 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:04.756256    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:04.756729 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:04.756299    4473 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:53:04.756729 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:04.756385    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs podName:dbf75704-af7b-44f6-bfcb-b2b2e943f5e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:06.756364368 +0000 UTC m=+4.992573710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs") pod "network-metrics-daemon-kprst" (UID: "dbf75704-af7b-44f6-bfcb-b2b2e943f5e6") : secret "metrics-daemon-secret" not found
Oct 30 21:53:04.756729 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:04.756389    4473 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:53:04.756729 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:04.756447    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls podName:18aafbda-138f-40ee-9b40-9e1b76163b98 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:06.756430247 +0000 UTC m=+4.992639583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls") pod "dns-default-wx8d9" (UID: "18aafbda-138f-40ee-9b40-9e1b76163b98") : secret "dns-default-metrics-tls" not found
Oct 30 21:53:04.756729 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:04.756390    4473 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:53:04.756729 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:04.756493    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls podName:b4d48390-828e-4c3e-91e8-44264a171fd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:06.756483235 +0000 UTC m=+4.992692561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nf7t4" (UID: "b4d48390-828e-4c3e-91e8-44264a171fd3") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:53:04.788486 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:04.788451    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:04.928792 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:04.928725    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w68zw-mzlkv" event={"ID":"8faee8d2-4d27-4709-af52-4b5ef32592f6","Type":"ContainerStarted","Data":"41589b06dc449203e8d4f4eb7548f30fd0d475474b4fe41974f5835411426ad5"}
Oct 30 21:53:05.131218 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:05.131138    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:53:06.770592 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:06.770549    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:06.771049 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:06.770609    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:53:06.771049 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:06.770688    4473 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:53:06.771049 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:06.770711    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:53:06.771049 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:06.770734    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs podName:dbf75704-af7b-44f6-bfcb-b2b2e943f5e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:10.770722248 +0000 UTC m=+9.006931571 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs") pod "network-metrics-daemon-kprst" (UID: "dbf75704-af7b-44f6-bfcb-b2b2e943f5e6") : secret "metrics-daemon-secret" not found
Oct 30 21:53:06.771049 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:06.770689    4473 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:53:06.771049 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:06.770821    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls podName:b4d48390-828e-4c3e-91e8-44264a171fd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:10.770806231 +0000 UTC m=+9.007015555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nf7t4" (UID: "b4d48390-828e-4c3e-91e8-44264a171fd3") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:53:06.771049 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:06.770765    4473 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:53:06.771049 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:06.770862    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls podName:18aafbda-138f-40ee-9b40-9e1b76163b98 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:10.770853475 +0000 UTC m=+9.007062839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls") pod "dns-default-wx8d9" (UID: "18aafbda-138f-40ee-9b40-9e1b76163b98") : secret "dns-default-metrics-tls" not found
Oct 30 21:53:06.936216 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:06.936177    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-72s7t" event={"ID":"2aa2d0cb-19e6-4d3a-8aae-7760c255bef3","Type":"ContainerStarted","Data":"2ed39fa6270aebd338127db811fa7c29e6fb8c0a93ec9ad75fff87429a60b335"}
Oct 30 21:53:07.402498 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:07.402452    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:07.682015 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:07.681910    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:07.839097 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:07.839061    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:07.998102 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:07.998009    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:08.130722 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:08.130694    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:08.130908 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:08.130897    4473 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:53:08.153801 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:08.153772    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:08.299759 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:08.299683    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8xr5"
Oct 30 21:53:08.321975 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:08.321933    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:08.591005 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:08.590922    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x2gfj_53d63a68-43fa-4e92-9a39-541cf8983a89/dns-node-resolver/0.log"
Oct 30 21:53:08.623011 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:08.622981    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:08.775476 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:08.775444    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:09.435552 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:09.435484    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:09.591571 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:09.591543    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pcjh8_f60ffcff-4c40-4ae9-b5bd-2e23bf3bd098/node-ca/0.log"
Oct 30 21:53:09.828489 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:09.828379    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:09.991789 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:09.991753    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8w5d6_176bb59e-df7f-4b7a-8335-4c23f633ac1f/serve-healthcheck-canary/0.log"
Oct 30 21:53:10.023833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:10.023804    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:10.223964 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:10.223928    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:10.592658 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:10.592568    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-dlnp8_f37fd220-ad2c-4fd0-9d24-6b9733ce7d28/migrator/0.log"
Oct 30 21:53:10.682321 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:10.682295    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:10.800074 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:10.800036    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:53:10.800074 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:10.800076    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:53:10.800291 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:10.800118    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:10.800291 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:10.800196    4473 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:53:10.800291 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:10.800224    4473 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:53:10.800291 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:10.800269    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs podName:dbf75704-af7b-44f6-bfcb-b2b2e943f5e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:18.800252819 +0000 UTC m=+17.036462144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs") pod "network-metrics-daemon-kprst" (UID: "dbf75704-af7b-44f6-bfcb-b2b2e943f5e6") : secret "metrics-daemon-secret" not found
Oct 30 21:53:10.800291 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:10.800291    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls podName:b4d48390-828e-4c3e-91e8-44264a171fd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:18.800276582 +0000 UTC m=+17.036485909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nf7t4" (UID: "b4d48390-828e-4c3e-91e8-44264a171fd3") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:53:10.800503 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:10.800482    4473 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:53:10.802833 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:10.802803    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18aafbda-138f-40ee-9b40-9e1b76163b98-metrics-tls\") pod \"dns-default-wx8d9\" (UID: \"18aafbda-138f-40ee-9b40-9e1b76163b98\") " pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:53:11.089260 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:11.089232    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:53:11.422798 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:11.422765    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:11.460177 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:11.460149    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wx8d9"]
Oct 30 21:53:11.463220 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:11.463180    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18aafbda_138f_40ee_9b40_9e1b76163b98.slice/crio-1bd4ef0c1c9fba6d613703f8935ccaea2454a5996d007520c09bd0f288a22941 WatchSource:0}: Error finding container 1bd4ef0c1c9fba6d613703f8935ccaea2454a5996d007520c09bd0f288a22941: Status 404 returned error can't find the container with id 1bd4ef0c1c9fba6d613703f8935ccaea2454a5996d007520c09bd0f288a22941
Oct 30 21:53:11.796055 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:11.795969    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:11.948813 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:11.948785    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wx8d9" event={"ID":"18aafbda-138f-40ee-9b40-9e1b76163b98","Type":"ContainerStarted","Data":"1bd4ef0c1c9fba6d613703f8935ccaea2454a5996d007520c09bd0f288a22941"}
Oct 30 21:53:11.954733 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:11.954702    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:12.117791 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:12.117712    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:12.273526 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:12.273490    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:12.631331 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:12.631283    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:13.220898 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:13.220862    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:13.443645 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:13.443602    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:13.600555 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:13.600434    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:13.825987 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:13.825912    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:13.961459 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:13.961423    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wx8d9" event={"ID":"18aafbda-138f-40ee-9b40-9e1b76163b98","Type":"ContainerStarted","Data":"644673fae2480ce66487240317d261820e012b2a71b18d991e8ccc389facb462"}
Oct 30 21:53:13.961616 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:13.961473    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wx8d9" event={"ID":"18aafbda-138f-40ee-9b40-9e1b76163b98","Type":"ContainerStarted","Data":"67dc13931970a82f76ee01248eeadab4e6e11f9cc65e7d31d7a24e3f5c9b105f"}
Oct 30 21:53:13.961616 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:13.961563    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:53:14.038355 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:14.038323    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:14.224011 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:14.223896    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:14.631061 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:14.630984    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:15.114876 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:15.114846    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2mskp"
Oct 30 21:53:15.455449 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:15.455382    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:15.609671 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:15.609632    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:15.780221 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:15.780137    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:15.933147 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:15.933114    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:16.084441 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:16.084362    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:16.238072 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:16.238043    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:16.402149 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:16.402086    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:16.549968 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:16.549924    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:17.109961 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:17.109910    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:17.271533 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:17.271497    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:17.643649 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:17.643614    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:17.833221 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:17.833191    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:18.021576 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:18.021543    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:18.226406 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:18.226372    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:18.432233 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:18.432202    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:18.582641 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:18.582606    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:18.856509 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:18.856435    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:53:18.856509 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:18.856480    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:18.858941 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:18.858913    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d48390-828e-4c3e-91e8-44264a171fd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nf7t4\" (UID: \"b4d48390-828e-4c3e-91e8-44264a171fd3\") " pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:18.859068 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:18.859010    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf75704-af7b-44f6-bfcb-b2b2e943f5e6-metrics-certs\") pod \"network-metrics-daemon-kprst\" (UID: \"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6\") " pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:53:18.887067 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:18.887030    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kprst"
Oct 30 21:53:18.889753 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:18.889732    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nf7t4"
Oct 30 21:53:19.313373 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:19.313350    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kprst"]
Oct 30 21:53:19.315487 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:19.315462    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf75704_af7b_44f6_bfcb_b2b2e943f5e6.slice/crio-93228bb130d6f5426b5a7912cc754509451d6a652d27142ddda3639fa3cd7d40 WatchSource:0}: Error finding container 93228bb130d6f5426b5a7912cc754509451d6a652d27142ddda3639fa3cd7d40: Status 404 returned error can't find the container with id 93228bb130d6f5426b5a7912cc754509451d6a652d27142ddda3639fa3cd7d40
Oct 30 21:53:19.364944 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:19.364914    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nf7t4"]
Oct 30 21:53:19.367835 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:19.367814    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d48390_828e_4c3e_91e8_44264a171fd3.slice/crio-601ae27a9c0ddb7ff650993e5372865f3ffe5092ac8f4bd9d7a22242349aa088 WatchSource:0}: Error finding container 601ae27a9c0ddb7ff650993e5372865f3ffe5092ac8f4bd9d7a22242349aa088: Status 404 returned error can't find the container with id 601ae27a9c0ddb7ff650993e5372865f3ffe5092ac8f4bd9d7a22242349aa088
Oct 30 21:53:19.984137 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:19.984100    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kprst" event={"ID":"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6","Type":"ContainerStarted","Data":"93228bb130d6f5426b5a7912cc754509451d6a652d27142ddda3639fa3cd7d40"}
Oct 30 21:53:19.985492 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:19.985470    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nf7t4" event={"ID":"b4d48390-828e-4c3e-91e8-44264a171fd3","Type":"ContainerStarted","Data":"dc257966f61f246d16da69377834c4e5159884d85bd68fe285787b4c89bb04e8"}
Oct 30 21:53:19.985572 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:19.985515    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nf7t4" event={"ID":"b4d48390-828e-4c3e-91e8-44264a171fd3","Type":"ContainerStarted","Data":"601ae27a9c0ddb7ff650993e5372865f3ffe5092ac8f4bd9d7a22242349aa088"}
Oct 30 21:53:20.990236 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:20.990198    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kprst" event={"ID":"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6","Type":"ContainerStarted","Data":"0d4f9287991c7308a900c3f029d847bda9e58ce3721e478ab9544601d443a8c1"}
Oct 30 21:53:20.990672 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:20.990261    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kprst" event={"ID":"dbf75704-af7b-44f6-bfcb-b2b2e943f5e6","Type":"ContainerStarted","Data":"fbcad19f6b4fdccc8f506432c3f82df502309e312d5bf31ec666c6cf1cba6b58"}
Oct 30 21:53:20.991745 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:20.991724    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nf7t4" event={"ID":"b4d48390-828e-4c3e-91e8-44264a171fd3","Type":"ContainerStarted","Data":"05d37d7f04fe5b39ce14b203bc19d37b7b9f6391df99785c412dfdd3ab7b184c"}
Oct 30 21:53:22.840806 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:22.840769    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:22.996360 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:22.996324    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:23.000356 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.000331    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nf7t4" event={"ID":"b4d48390-828e-4c3e-91e8-44264a171fd3","Type":"ContainerStarted","Data":"6783b9539157f029ec9d9125d16053e5ca3f2c5a2b1546c58cd50ffa817f1723"}
Oct 30 21:53:23.172665 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.172634    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:23.318945 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.318906    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-bj7lq"]
Oct 30 21:53:23.321059 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.321034    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-bj7lq"
Oct 30 21:53:23.381619 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.381596    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:53:23.381805 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.381683    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:53:23.383361 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.383343    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-kt8qx\""
Oct 30 21:53:23.385138 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.385115    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-bj7lq"]
Oct 30 21:53:23.394660 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.394631    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99a25eed-4eb9-4f87-8520-139ea86cf7ef-tmp\") pod \"downloads-6dd789c8b7-bj7lq\" (UID: \"99a25eed-4eb9-4f87-8520-139ea86cf7ef\") " pod="openshift-console/downloads-6dd789c8b7-bj7lq"
Oct 30 21:53:23.394918 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.394664    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tmk5\" (UniqueName: \"kubernetes.io/projected/99a25eed-4eb9-4f87-8520-139ea86cf7ef-kube-api-access-8tmk5\") pod \"downloads-6dd789c8b7-bj7lq\" (UID: \"99a25eed-4eb9-4f87-8520-139ea86cf7ef\") " pod="openshift-console/downloads-6dd789c8b7-bj7lq"
Oct 30 21:53:23.461275 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.461198    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:23.495820 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.495792    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99a25eed-4eb9-4f87-8520-139ea86cf7ef-tmp\") pod \"downloads-6dd789c8b7-bj7lq\" (UID: \"99a25eed-4eb9-4f87-8520-139ea86cf7ef\") " pod="openshift-console/downloads-6dd789c8b7-bj7lq"
Oct 30 21:53:23.495930 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.495829    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tmk5\" (UniqueName: \"kubernetes.io/projected/99a25eed-4eb9-4f87-8520-139ea86cf7ef-kube-api-access-8tmk5\") pod \"downloads-6dd789c8b7-bj7lq\" (UID: \"99a25eed-4eb9-4f87-8520-139ea86cf7ef\") " pod="openshift-console/downloads-6dd789c8b7-bj7lq"
Oct 30 21:53:23.496193 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.496173    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99a25eed-4eb9-4f87-8520-139ea86cf7ef-tmp\") pod \"downloads-6dd789c8b7-bj7lq\" (UID: \"99a25eed-4eb9-4f87-8520-139ea86cf7ef\") " pod="openshift-console/downloads-6dd789c8b7-bj7lq"
Oct 30 21:53:23.555421 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.555388    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tmk5\" (UniqueName: \"kubernetes.io/projected/99a25eed-4eb9-4f87-8520-139ea86cf7ef-kube-api-access-8tmk5\") pod \"downloads-6dd789c8b7-bj7lq\" (UID: \"99a25eed-4eb9-4f87-8520-139ea86cf7ef\") " pod="openshift-console/downloads-6dd789c8b7-bj7lq"
Oct 30 21:53:23.630635 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.630601    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-bj7lq"
Oct 30 21:53:23.679780 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.679116    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:23.844371 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.844291    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:23.967187 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:23.967159    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wx8d9"
Oct 30 21:53:24.065825 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:24.065777    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:24.074002 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:24.073979    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xmc8p"]
Oct 30 21:53:24.076132 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:24.076115    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xmc8p"
Oct 30 21:53:24.099615 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:24.099564    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5762586d-f5b1-4339-9c80-3e703eb00133-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-xmc8p\" (UID: \"5762586d-f5b1-4339-9c80-3e703eb00133\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xmc8p"
Oct 30 21:53:24.102407 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:24.102388    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-bj7lq"]
Oct 30 21:53:24.127672 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:24.127641    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a25eed_4eb9_4f87_8520_139ea86cf7ef.slice/crio-7cde8fec7d8344e660d84be6d36b6000a953757d5dfad9631f93c857ca3795a7 WatchSource:0}: Error finding container 7cde8fec7d8344e660d84be6d36b6000a953757d5dfad9631f93c857ca3795a7: Status 404 returned error can't find the container with id 7cde8fec7d8344e660d84be6d36b6000a953757d5dfad9631f93c857ca3795a7
Oct 30 21:53:24.129612 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:24.129586    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gp5mj\""
Oct 30 21:53:24.129722 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:24.129663    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 21:53:24.135708 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:24.135686    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xmc8p"]
Oct 30 21:53:24.200187 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:24.200148    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5762586d-f5b1-4339-9c80-3e703eb00133-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-xmc8p\" (UID: \"5762586d-f5b1-4339-9c80-3e703eb00133\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xmc8p"
Oct 30 21:53:24.202702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:24.202678    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5762586d-f5b1-4339-9c80-3e703eb00133-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-xmc8p\" (UID: \"5762586d-f5b1-4339-9c80-3e703eb00133\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xmc8p"
Oct 30 21:53:24.222211 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:24.222180    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:24.385876 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:24.385792    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xmc8p"
Oct 30 21:53:24.753346 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:24.753308    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xmc8p"]
Oct 30 21:53:24.756132 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:24.756105    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5762586d_f5b1_4339_9c80_3e703eb00133.slice/crio-3d53966455fbeff0b70fce9ac865d021254867d7233c8f8ca3729447c55c57f4 WatchSource:0}: Error finding container 3d53966455fbeff0b70fce9ac865d021254867d7233c8f8ca3729447c55c57f4: Status 404 returned error can't find the container with id 3d53966455fbeff0b70fce9ac865d021254867d7233c8f8ca3729447c55c57f4
Oct 30 21:53:25.008006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:25.007899    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-bj7lq" event={"ID":"99a25eed-4eb9-4f87-8520-139ea86cf7ef","Type":"ContainerStarted","Data":"7cde8fec7d8344e660d84be6d36b6000a953757d5dfad9631f93c857ca3795a7"}
Oct 30 21:53:25.008994 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:25.008975    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xmc8p" event={"ID":"5762586d-f5b1-4339-9c80-3e703eb00133","Type":"ContainerStarted","Data":"3d53966455fbeff0b70fce9ac865d021254867d7233c8f8ca3729447c55c57f4"}
Oct 30 21:53:25.895527 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:25.895498    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-84ff54fd5b-hzdg8"]
Oct 30 21:53:25.898560 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:25.898535    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.002290 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.002254    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 21:53:26.002472 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.002254    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 21:53:26.002472 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.002453    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 21:53:26.002707 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.002692    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 21:53:26.002976 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.002935    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 21:53:26.003058 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.003011    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bxhgx\""
Oct 30 21:53:26.003773 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.003747    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84ff54fd5b-hzdg8"]
Oct 30 21:53:26.014511 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.014480    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c869caa-0bd6-4211-824a-ddd3c8e57a95-tmp\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.014511 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.014511    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7mll\" (UniqueName: \"kubernetes.io/projected/5c869caa-0bd6-4211-824a-ddd3c8e57a95-kube-api-access-d7mll\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.014945 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.014530    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-service-ca\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.014945 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.014551    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-config\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.014945 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.014662    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-oauth-serving-cert\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.014945 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.014690    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-serving-cert\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.014945 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.014711    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-oauth-config\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.115842 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.115798    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c869caa-0bd6-4211-824a-ddd3c8e57a95-tmp\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.116035 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.115865    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7mll\" (UniqueName: \"kubernetes.io/projected/5c869caa-0bd6-4211-824a-ddd3c8e57a95-kube-api-access-d7mll\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.116035 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.115895    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-service-ca\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.116035 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.115917    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-config\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.116035 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.115940    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-oauth-serving-cert\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.116035 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.115987    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-serving-cert\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.116035 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.116008    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-oauth-config\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.116260 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.116231    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c869caa-0bd6-4211-824a-ddd3c8e57a95-tmp\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.116836 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.116813    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-config\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.117165 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.117145    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-service-ca\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.117355 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.117299    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-oauth-serving-cert\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.118975 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.118930    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-oauth-config\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.119274 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.119250    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-serving-cert\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.175631 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.175558    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7mll\" (UniqueName: \"kubernetes.io/projected/5c869caa-0bd6-4211-824a-ddd3c8e57a95-kube-api-access-d7mll\") pod \"console-84ff54fd5b-hzdg8\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") " pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.212361 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.212323    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:26.592704 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:26.592665    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84ff54fd5b-hzdg8"]
Oct 30 21:53:26.627087 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:26.627060    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c869caa_0bd6_4211_824a_ddd3c8e57a95.slice/crio-e5c1a485cb9aadf8a79e53feac2b77a41427bd2b411c924356126f06318a695a WatchSource:0}: Error finding container e5c1a485cb9aadf8a79e53feac2b77a41427bd2b411c924356126f06318a695a: Status 404 returned error can't find the container with id e5c1a485cb9aadf8a79e53feac2b77a41427bd2b411c924356126f06318a695a
Oct 30 21:53:27.021192 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:27.021141    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xmc8p" event={"ID":"5762586d-f5b1-4339-9c80-3e703eb00133","Type":"ContainerStarted","Data":"24a46d0fcc5ede6dc1578ad562006524e3d0a59ad133db66df3764be85e44987"}
Oct 30 21:53:27.021672 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:27.021242    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xmc8p"
Oct 30 21:53:27.022863 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:27.022834    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84ff54fd5b-hzdg8" event={"ID":"5c869caa-0bd6-4211-824a-ddd3c8e57a95","Type":"ContainerStarted","Data":"e5c1a485cb9aadf8a79e53feac2b77a41427bd2b411c924356126f06318a695a"}
Oct 30 21:53:27.026592 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:27.026570    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xmc8p"
Oct 30 21:53:27.257541 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:27.257486    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xmc8p" podStartSLOduration=1.500551676 podStartE2EDuration="3.257470782s" totalImagesPullingTime="1.756919106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:27.140974948 +0000 UTC m=+25.377184295" watchObservedRunningTime="2026-10-30 21:53:27.257470782 +0000 UTC m=+25.493680129"
Oct 30 21:53:28.279628 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:28.279573    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:28.459434 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:28.459397    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:28.615210 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:28.615130    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:28.792488 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:28.792456    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:30.308570 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:30.308530    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:30.465360 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:30.465324    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:31.084656 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:31.084625    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:32.041435 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:32.041394    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84ff54fd5b-hzdg8" event={"ID":"5c869caa-0bd6-4211-824a-ddd3c8e57a95","Type":"ContainerStarted","Data":"46dfbcc342f2ce832fd4219d0cac94c3a9644a82f7ae15b718e4dd00a1a82c83"}
Oct 30 21:53:32.109479 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:32.109431    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:32.184816 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:32.184751    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-84ff54fd5b-hzdg8" podStartSLOduration=2.7610267889999998 podStartE2EDuration="7.184731629s" totalImagesPullingTime="4.42370484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:32.184712495 +0000 UTC m=+30.420921839" watchObservedRunningTime="2026-10-30 21:53:32.184731629 +0000 UTC m=+30.420940973"
Oct 30 21:53:32.292281 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:32.292199    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:33.203499 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.203457    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:33.355296 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.355258    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:33.696198 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.696166    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:33.805985 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.805928    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-674b886b45-85zrp"]
Oct 30 21:53:33.837491 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.837454    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.851067 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.851040    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:33.870540 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.870515    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-674b886b45-85zrp"]
Oct 30 21:53:33.873978 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.873923    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-serving-cert\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.874091 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.873981    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a59ce7d-37c9-451f-90be-de729bab3bb3-tmp\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.874091 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.874044    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-config\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.874091 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.874070    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-oauth-serving-cert\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.874228 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.874152    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-service-ca\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.874228 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.874193    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-trusted-ca-bundle\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.874293 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.874273    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg9hx\" (UniqueName: \"kubernetes.io/projected/5a59ce7d-37c9-451f-90be-de729bab3bb3-kube-api-access-kg9hx\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.874330 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.874303    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-oauth-config\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.975661 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.975576    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-oauth-config\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.975661 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.975613    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-serving-cert\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.975661 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.975642    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a59ce7d-37c9-451f-90be-de729bab3bb3-tmp\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.975927 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.975671    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-config\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.975927 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.975708    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-oauth-serving-cert\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.975927 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.975732    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-service-ca\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.975927 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.975765    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-trusted-ca-bundle\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.975927 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.975819    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg9hx\" (UniqueName: \"kubernetes.io/projected/5a59ce7d-37c9-451f-90be-de729bab3bb3-kube-api-access-kg9hx\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.976216 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.976164    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a59ce7d-37c9-451f-90be-de729bab3bb3-tmp\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.976624 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.976588    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-config\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.976624 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.976611    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-oauth-serving-cert\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.976798 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.976710    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-service-ca\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.978806 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.978781    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-serving-cert\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:33.980353 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:33.980329    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-oauth-config\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:34.022291 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:34.022249    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 21:53:34.027458 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:34.027430    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-trusted-ca-bundle\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:34.045079 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:34.045027    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg9hx\" (UniqueName: \"kubernetes.io/projected/5a59ce7d-37c9-451f-90be-de729bab3bb3-kube-api-access-kg9hx\") pod \"console-674b886b45-85zrp\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") " pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:34.162037 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:34.161998    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:34.369597 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:34.369540    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:34.548961 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:34.548834    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:34.567505 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:34.567421    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-674b886b45-85zrp"]
Oct 30 21:53:34.603594 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:34.603561    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a59ce7d_37c9_451f_90be_de729bab3bb3.slice/crio-0ac3ac91d59a8b935af12fce7dc3c0c8ca07075ff8ec32ae467aee10abf6b14d WatchSource:0}: Error finding container 0ac3ac91d59a8b935af12fce7dc3c0c8ca07075ff8ec32ae467aee10abf6b14d: Status 404 returned error can't find the container with id 0ac3ac91d59a8b935af12fce7dc3c0c8ca07075ff8ec32ae467aee10abf6b14d
Oct 30 21:53:34.703732 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:34.703703    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:34.912190 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:34.912149    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:35.054726 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.054236    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-674b886b45-85zrp" event={"ID":"5a59ce7d-37c9-451f-90be-de729bab3bb3","Type":"ContainerStarted","Data":"134e7069ffb9b8aa8448680cc66e1aba618f18134eac8f999ed20e9f709d2cab"}
Oct 30 21:53:35.054726 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.054294    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-674b886b45-85zrp" event={"ID":"5a59ce7d-37c9-451f-90be-de729bab3bb3","Type":"ContainerStarted","Data":"0ac3ac91d59a8b935af12fce7dc3c0c8ca07075ff8ec32ae467aee10abf6b14d"}
Oct 30 21:53:35.112400 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.112033    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:35.134027 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.133997    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b5h24"
Oct 30 21:53:35.187992 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.187877    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-674b886b45-85zrp" podStartSLOduration=2.187865771 podStartE2EDuration="2.187865771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:35.187299755 +0000 UTC m=+33.423509099" watchObservedRunningTime="2026-10-30 21:53:35.187865771 +0000 UTC m=+33.424075116"
Oct 30 21:53:35.268802 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.268762    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:35.285660 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.285623    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wfk5m"]
Oct 30 21:53:35.308324 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.308288    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"]
Oct 30 21:53:35.308503 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.308436    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.326498 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.326475    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.363427 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.363395    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dq6s5\""
Oct 30 21:53:35.386451 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.386415    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/290ee42e-6821-4813-bb88-557f4e99946e-node-exporter-tls\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.386914 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.386466    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxx6f\" (UniqueName: \"kubernetes.io/projected/290ee42e-6821-4813-bb88-557f4e99946e-kube-api-access-xxx6f\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.386914 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.386494    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jh9b\" (UniqueName: \"kubernetes.io/projected/8ca11674-8465-4bc9-b1a9-e2b99135d080-kube-api-access-6jh9b\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.386914 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.386618    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/290ee42e-6821-4813-bb88-557f4e99946e-root\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.386914 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.386651    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ca11674-8465-4bc9-b1a9-e2b99135d080-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.386914 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.386740    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/290ee42e-6821-4813-bb88-557f4e99946e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.386914 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.386802    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ca11674-8465-4bc9-b1a9-e2b99135d080-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.386914 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.386854    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/290ee42e-6821-4813-bb88-557f4e99946e-sys\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.387245 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.386926    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/290ee42e-6821-4813-bb88-557f4e99946e-node-exporter-textfile\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.387245 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.386976    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/290ee42e-6821-4813-bb88-557f4e99946e-metrics-client-ca\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.387245 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.387011    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ca11674-8465-4bc9-b1a9-e2b99135d080-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.387245 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.387041    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/290ee42e-6821-4813-bb88-557f4e99946e-node-exporter-wtmp\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.387245 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.387067    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/290ee42e-6821-4813-bb88-557f4e99946e-node-exporter-accelerators-collector-config\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.387245 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.387108    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ca11674-8465-4bc9-b1a9-e2b99135d080-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.387245 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.387136    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ca11674-8465-4bc9-b1a9-e2b99135d080-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.387639 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.387613    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"]
Oct 30 21:53:35.431990 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.431928    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:35.461488 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.461407    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:53:35.461640 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.461578    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:53:35.461640 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.461631    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 21:53:35.461720 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.461636    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:53:35.461720 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.461634    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:53:35.461782 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.461765    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xx89s\""
Oct 30 21:53:35.487718 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.487686    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ca11674-8465-4bc9-b1a9-e2b99135d080-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.487718 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.487729    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/290ee42e-6821-4813-bb88-557f4e99946e-node-exporter-tls\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.487967 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.487777    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxx6f\" (UniqueName: \"kubernetes.io/projected/290ee42e-6821-4813-bb88-557f4e99946e-kube-api-access-xxx6f\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.487967 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.487797    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jh9b\" (UniqueName: \"kubernetes.io/projected/8ca11674-8465-4bc9-b1a9-e2b99135d080-kube-api-access-6jh9b\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.487967 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.487824    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/290ee42e-6821-4813-bb88-557f4e99946e-root\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.487967 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.487844    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ca11674-8465-4bc9-b1a9-e2b99135d080-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.487967 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.487869    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/290ee42e-6821-4813-bb88-557f4e99946e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.487967 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.487903    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ca11674-8465-4bc9-b1a9-e2b99135d080-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.487967 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.487924    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/290ee42e-6821-4813-bb88-557f4e99946e-sys\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.488325 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.487979    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/290ee42e-6821-4813-bb88-557f4e99946e-node-exporter-textfile\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.488325 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.488005    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/290ee42e-6821-4813-bb88-557f4e99946e-metrics-client-ca\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.488325 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.488036    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ca11674-8465-4bc9-b1a9-e2b99135d080-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.488325 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.488055    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/290ee42e-6821-4813-bb88-557f4e99946e-node-exporter-wtmp\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.488325 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.488072    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/290ee42e-6821-4813-bb88-557f4e99946e-node-exporter-accelerators-collector-config\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.488325 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.488099    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ca11674-8465-4bc9-b1a9-e2b99135d080-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.488325 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.488288    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/290ee42e-6821-4813-bb88-557f4e99946e-sys\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.488583 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.488561    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/290ee42e-6821-4813-bb88-557f4e99946e-node-exporter-textfile\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.488962 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.488921    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ca11674-8465-4bc9-b1a9-e2b99135d080-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.489092 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.489068    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/290ee42e-6821-4813-bb88-557f4e99946e-root\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.489092 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.489089    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/290ee42e-6821-4813-bb88-557f4e99946e-node-exporter-wtmp\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.491057 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.491034    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ca11674-8465-4bc9-b1a9-e2b99135d080-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.491568 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.491551    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/290ee42e-6821-4813-bb88-557f4e99946e-node-exporter-tls\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.491875 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.491850    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/290ee42e-6821-4813-bb88-557f4e99946e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.506391 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.506364    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:53:35.506505 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.506413    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 21:53:35.506505 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.506426    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 21:53:35.506505 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.506458    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:53:35.508868 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.508848    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ca11674-8465-4bc9-b1a9-e2b99135d080-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.509310 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.509284    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/290ee42e-6821-4813-bb88-557f4e99946e-metrics-client-ca\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.509651 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.509632    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/290ee42e-6821-4813-bb88-557f4e99946e-node-exporter-accelerators-collector-config\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.509789 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.509766    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ca11674-8465-4bc9-b1a9-e2b99135d080-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.510811 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.510794    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ca11674-8465-4bc9-b1a9-e2b99135d080-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.547807 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.547774    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jh9b\" (UniqueName: \"kubernetes.io/projected/8ca11674-8465-4bc9-b1a9-e2b99135d080-kube-api-access-6jh9b\") pod \"kube-state-metrics-5dc8d7d89b-rfl76\" (UID: \"8ca11674-8465-4bc9-b1a9-e2b99135d080\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.548204 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.548183    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxx6f\" (UniqueName: \"kubernetes.io/projected/290ee42e-6821-4813-bb88-557f4e99946e-kube-api-access-xxx6f\") pod \"node-exporter-wfk5m\" (UID: \"290ee42e-6821-4813-bb88-557f4e99946e\") " pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.658987 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.658934    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:35.666828 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.666797    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wfk5m"
Oct 30 21:53:35.709077 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:35.709035    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290ee42e_6821_4813_bb88_557f4e99946e.slice/crio-3a0158f15c0d7b641bf5f4178653a89f2791fd62bc71b133c93391d3fb673249 WatchSource:0}: Error finding container 3a0158f15c0d7b641bf5f4178653a89f2791fd62bc71b133c93391d3fb673249: Status 404 returned error can't find the container with id 3a0158f15c0d7b641bf5f4178653a89f2791fd62bc71b133c93391d3fb673249
Oct 30 21:53:35.719252 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.719227    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"
Oct 30 21:53:35.819847 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:35.819818    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:36.060516 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:36.060430    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wfk5m" event={"ID":"290ee42e-6821-4813-bb88-557f4e99946e","Type":"ContainerStarted","Data":"3a0158f15c0d7b641bf5f4178653a89f2791fd62bc71b133c93391d3fb673249"}
Oct 30 21:53:36.174008 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:36.173895    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76"]
Oct 30 21:53:36.176739 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:36.176699    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca11674_8465_4bc9_b1a9_e2b99135d080.slice/crio-8f2c3a7eed1bf77c97bf0b9c83a39f6f7917b1836d517e206d610bf094abffe5 WatchSource:0}: Error finding container 8f2c3a7eed1bf77c97bf0b9c83a39f6f7917b1836d517e206d610bf094abffe5: Status 404 returned error can't find the container with id 8f2c3a7eed1bf77c97bf0b9c83a39f6f7917b1836d517e206d610bf094abffe5
Oct 30 21:53:36.213529 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:36.213494    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:36.213688 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:36.213559    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:36.218436 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:36.218411    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:36.745478 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:36.745444    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:36.918702 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:36.918664    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:37.066444 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:37.066371    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76" event={"ID":"8ca11674-8465-4bc9-b1a9-e2b99135d080","Type":"ContainerStarted","Data":"8f2c3a7eed1bf77c97bf0b9c83a39f6f7917b1836d517e206d610bf094abffe5"}
Oct 30 21:53:37.071649 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:37.071622    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:53:37.561080 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:37.561040    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:37.962092 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:37.962056    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-c57bb4c94-x2drs"]
Oct 30 21:53:37.964521 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:37.964495    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:37.996170 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:37.996134    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c57bb4c94-x2drs"]
Oct 30 21:53:38.010151 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.010114    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-grpc-tls\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.010316 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.010182    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.010316 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.010215    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.010316 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.010242    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b599ef14-1c28-4c77-adbc-cb007513d00d-metrics-client-ca\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.010316 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.010287    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b599ef14-1c28-4c77-adbc-cb007513d00d-tmp\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.010316 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.010313    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx4cl\" (UniqueName: \"kubernetes.io/projected/b599ef14-1c28-4c77-adbc-cb007513d00d-kube-api-access-bx4cl\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.010544 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.010359    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-thanos-querier-tls\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.010544 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.010395    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.010544 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.010430    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.045699 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.045340    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 21:53:38.045699 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.045389    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 21:53:38.045699 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.045545    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 21:53:38.045699 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.045566    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-cbmvl\""
Oct 30 21:53:38.045699 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.045576    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 21:53:38.045699 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.045652    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7g1atro5fkbii\""
Oct 30 21:53:38.045699 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.045653    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 21:53:38.072527 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.072494    4473 generic.go:356] "Generic (PLEG): container finished" podID="290ee42e-6821-4813-bb88-557f4e99946e" containerID="e77221ccb7b651f3ce93f1b4bfa946e050bb15aa48536b87a867b7343bae97d0" exitCode=0
Oct 30 21:53:38.072697 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.072577    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wfk5m" event={"ID":"290ee42e-6821-4813-bb88-557f4e99946e","Type":"ContainerDied","Data":"e77221ccb7b651f3ce93f1b4bfa946e050bb15aa48536b87a867b7343bae97d0"}
Oct 30 21:53:38.111551 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.111523    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b599ef14-1c28-4c77-adbc-cb007513d00d-tmp\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.111694 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.111566    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx4cl\" (UniqueName: \"kubernetes.io/projected/b599ef14-1c28-4c77-adbc-cb007513d00d-kube-api-access-bx4cl\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.111694 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.111663    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-thanos-querier-tls\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.111818 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.111714    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.111818 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.111761    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.111985 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.111935    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b599ef14-1c28-4c77-adbc-cb007513d00d-tmp\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.112104 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.112079    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-grpc-tls\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.112172 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.112130    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.112172 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.112164    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.112267 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.112190    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b599ef14-1c28-4c77-adbc-cb007513d00d-metrics-client-ca\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.113393 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.113366    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b599ef14-1c28-4c77-adbc-cb007513d00d-metrics-client-ca\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.114717 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.114689    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-thanos-querier-tls\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.115509 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.115484    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.116284 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.116247    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-grpc-tls\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.116380 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.116325    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.117302 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.117280    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.118637 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.118612    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b599ef14-1c28-4c77-adbc-cb007513d00d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.170966 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.170928    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx4cl\" (UniqueName: \"kubernetes.io/projected/b599ef14-1c28-4c77-adbc-cb007513d00d-kube-api-access-bx4cl\") pod \"thanos-querier-c57bb4c94-x2drs\" (UID: \"b599ef14-1c28-4c77-adbc-cb007513d00d\") " pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.291761 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.291731    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:38.731728 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.731253    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c57bb4c94-x2drs"]
Oct 30 21:53:38.731882 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.731857    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:38.742015 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:38.734485    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb599ef14_1c28_4c77_adbc_cb007513d00d.slice/crio-ddd8b9caafd7316ce5f010f2936d0282f584af6ccec20776013ac78772f784d7 WatchSource:0}: Error finding container ddd8b9caafd7316ce5f010f2936d0282f584af6ccec20776013ac78772f784d7: Status 404 returned error can't find the container with id ddd8b9caafd7316ce5f010f2936d0282f584af6ccec20776013ac78772f784d7
Oct 30 21:53:38.894586 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:38.894548    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:39.079545 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:39.079462    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wfk5m" event={"ID":"290ee42e-6821-4813-bb88-557f4e99946e","Type":"ContainerStarted","Data":"bfedf66a7d93f2031fd88cf257effcfb69de4423dfb194a4d0b975bdb32d464d"}
Oct 30 21:53:39.079545 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:39.079524    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wfk5m" event={"ID":"290ee42e-6821-4813-bb88-557f4e99946e","Type":"ContainerStarted","Data":"5dad3b814a3b35ed2dbcf0f563d5b48a5ac91c79d0e0adcb90862148865094b0"}
Oct 30 21:53:39.082116 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:39.082085    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76" event={"ID":"8ca11674-8465-4bc9-b1a9-e2b99135d080","Type":"ContainerStarted","Data":"75c23dbf65ec0ba34ee8382ccf3d89b2602f4fd3d4bb2676c717fbaabc62491d"}
Oct 30 21:53:39.082278 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:39.082139    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76" event={"ID":"8ca11674-8465-4bc9-b1a9-e2b99135d080","Type":"ContainerStarted","Data":"d4f9d21c9726701033d6b55502b15c9553eac7c7c2e97722732b8a7f4afcff79"}
Oct 30 21:53:39.082278 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:39.082165    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76" event={"ID":"8ca11674-8465-4bc9-b1a9-e2b99135d080","Type":"ContainerStarted","Data":"9b92e0263df7e4077ee126a42f80655de62cf1a116111a8bf95d9b6e62672a87"}
Oct 30 21:53:39.083486 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:39.083453    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs" event={"ID":"b599ef14-1c28-4c77-adbc-cb007513d00d","Type":"ContainerStarted","Data":"ddd8b9caafd7316ce5f010f2936d0282f584af6ccec20776013ac78772f784d7"}
Oct 30 21:53:39.204365 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:39.204274    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wfk5m" podStartSLOduration=2.110047728 podStartE2EDuration="4.204257514s" totalImagesPullingTime="1.990165367s" totalInitContainerRuntime="104.044419ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:39.203991065 +0000 UTC m=+37.440200411" watchObservedRunningTime="2026-10-30 21:53:39.204257514 +0000 UTC m=+37.440466862"
Oct 30 21:53:39.324193 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:39.324140    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rfl76" podStartSLOduration=2.42297236 podStartE2EDuration="4.324124603s" totalImagesPullingTime="1.901152243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:39.323586067 +0000 UTC m=+37.559795413" watchObservedRunningTime="2026-10-30 21:53:39.324124603 +0000 UTC m=+37.560333948"
Oct 30 21:53:39.363120 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:39.363043    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:39.564214 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:39.564177    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:39.764726 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:39.764687    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:39.921893 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:39.921813    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:39.990473 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:39.990441    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-7l7hw"]
Oct 30 21:53:39.992583 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:39.992564    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-7l7hw"
Oct 30 21:53:40.029978 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.029870    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d494e8a5-5577-4ce2-af1f-cec218f37b50-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-7l7hw\" (UID: \"d494e8a5-5577-4ce2-af1f-cec218f37b50\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-7l7hw"
Oct 30 21:53:40.029978 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.029927    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct5dl\" (UniqueName: \"kubernetes.io/projected/d494e8a5-5577-4ce2-af1f-cec218f37b50-kube-api-access-ct5dl\") pod \"monitoring-plugin-77f4f76594-7l7hw\" (UID: \"d494e8a5-5577-4ce2-af1f-cec218f37b50\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-7l7hw"
Oct 30 21:53:40.052659 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.052624    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-7l7hw"]
Oct 30 21:53:40.063539 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.063509    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-k7b2j\""
Oct 30 21:53:40.063725 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.063618    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 21:53:40.132131 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.131006    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d494e8a5-5577-4ce2-af1f-cec218f37b50-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-7l7hw\" (UID: \"d494e8a5-5577-4ce2-af1f-cec218f37b50\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-7l7hw"
Oct 30 21:53:40.132131 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.131056    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct5dl\" (UniqueName: \"kubernetes.io/projected/d494e8a5-5577-4ce2-af1f-cec218f37b50-kube-api-access-ct5dl\") pod \"monitoring-plugin-77f4f76594-7l7hw\" (UID: \"d494e8a5-5577-4ce2-af1f-cec218f37b50\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-7l7hw"
Oct 30 21:53:40.133006 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.132946    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:40.134369 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.134349    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d494e8a5-5577-4ce2-af1f-cec218f37b50-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-7l7hw\" (UID: \"d494e8a5-5577-4ce2-af1f-cec218f37b50\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-7l7hw"
Oct 30 21:53:40.191048 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.191010    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct5dl\" (UniqueName: \"kubernetes.io/projected/d494e8a5-5577-4ce2-af1f-cec218f37b50-kube-api-access-ct5dl\") pod \"monitoring-plugin-77f4f76594-7l7hw\" (UID: \"d494e8a5-5577-4ce2-af1f-cec218f37b50\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-7l7hw"
Oct 30 21:53:40.226733 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.226699    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-9b5c6b74c-5w64w"]
Oct 30 21:53:40.228800 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.228773    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.283261 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.283184    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tcxml\""
Oct 30 21:53:40.283261 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.283228    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b834kh3s5cc7p\""
Oct 30 21:53:40.283491 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.283300    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 21:53:40.283491 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.283356    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 21:53:40.283491 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.283416    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 21:53:40.283491 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.283468    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 21:53:40.290926 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.290895    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9b5c6b74c-5w64w"]
Oct 30 21:53:40.306322 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.306062    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-7l7hw"
Oct 30 21:53:40.333301 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.333146    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3530dd2d-997b-48ca-9ffa-bfd102260075-secret-metrics-server-client-certs\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.333301 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.333215    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3530dd2d-997b-48ca-9ffa-bfd102260075-metrics-server-audit-profiles\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.333517 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.333323    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3530dd2d-997b-48ca-9ffa-bfd102260075-client-ca-bundle\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.333517 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.333368    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3530dd2d-997b-48ca-9ffa-bfd102260075-secret-metrics-server-tls\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.333517 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.333406    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8dvg\" (UniqueName: \"kubernetes.io/projected/3530dd2d-997b-48ca-9ffa-bfd102260075-kube-api-access-m8dvg\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.333517 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.333451    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3530dd2d-997b-48ca-9ffa-bfd102260075-audit-log\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.333517 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.333477    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3530dd2d-997b-48ca-9ffa-bfd102260075-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.434921 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.434882    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3530dd2d-997b-48ca-9ffa-bfd102260075-secret-metrics-server-client-certs\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.434921 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.434932    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3530dd2d-997b-48ca-9ffa-bfd102260075-metrics-server-audit-profiles\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.435158 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.434977    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3530dd2d-997b-48ca-9ffa-bfd102260075-client-ca-bundle\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.435158 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.435006    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3530dd2d-997b-48ca-9ffa-bfd102260075-secret-metrics-server-tls\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.435158 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.435045    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8dvg\" (UniqueName: \"kubernetes.io/projected/3530dd2d-997b-48ca-9ffa-bfd102260075-kube-api-access-m8dvg\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.435158 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.435096    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3530dd2d-997b-48ca-9ffa-bfd102260075-audit-log\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.435158 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.435126    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3530dd2d-997b-48ca-9ffa-bfd102260075-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.435861 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.435832    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3530dd2d-997b-48ca-9ffa-bfd102260075-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.436011 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.435988    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3530dd2d-997b-48ca-9ffa-bfd102260075-metrics-server-audit-profiles\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.436443 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.436407    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3530dd2d-997b-48ca-9ffa-bfd102260075-audit-log\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.438942 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.438918    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3530dd2d-997b-48ca-9ffa-bfd102260075-secret-metrics-server-client-certs\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.439071 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.439033    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3530dd2d-997b-48ca-9ffa-bfd102260075-client-ca-bundle\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.439123 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.439096    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3530dd2d-997b-48ca-9ffa-bfd102260075-secret-metrics-server-tls\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.494935 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.494899    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8dvg\" (UniqueName: \"kubernetes.io/projected/3530dd2d-997b-48ca-9ffa-bfd102260075-kube-api-access-m8dvg\") pod \"metrics-server-9b5c6b74c-5w64w\" (UID: \"3530dd2d-997b-48ca-9ffa-bfd102260075\") " pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.543383 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.543300    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:53:40.694213 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.693980    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-7l7hw"]
Oct 30 21:53:40.697241 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:40.697212    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd494e8a5_5577_4ce2_af1f_cec218f37b50.slice/crio-51889fe5188dd4019e9127fd04a3cf8d1ecd1fefa28d55003d75f2d562cd5323 WatchSource:0}: Error finding container 51889fe5188dd4019e9127fd04a3cf8d1ecd1fefa28d55003d75f2d562cd5323: Status 404 returned error can't find the container with id 51889fe5188dd4019e9127fd04a3cf8d1ecd1fefa28d55003d75f2d562cd5323
Oct 30 21:53:40.927419 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:40.927387    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9b5c6b74c-5w64w"]
Oct 30 21:53:40.932530 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:40.932496    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3530dd2d_997b_48ca_9ffa_bfd102260075.slice/crio-96f045eacc1f987b44a5c52a862a3b5763d4f49c94d40f384972a44af86cfccc WatchSource:0}: Error finding container 96f045eacc1f987b44a5c52a862a3b5763d4f49c94d40f384972a44af86cfccc: Status 404 returned error can't find the container with id 96f045eacc1f987b44a5c52a862a3b5763d4f49c94d40f384972a44af86cfccc
Oct 30 21:53:41.094313 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:41.094265    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w" event={"ID":"3530dd2d-997b-48ca-9ffa-bfd102260075","Type":"ContainerStarted","Data":"96f045eacc1f987b44a5c52a862a3b5763d4f49c94d40f384972a44af86cfccc"}
Oct 30 21:53:41.095801 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:41.095774    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-7l7hw" event={"ID":"d494e8a5-5577-4ce2-af1f-cec218f37b50","Type":"ContainerStarted","Data":"51889fe5188dd4019e9127fd04a3cf8d1ecd1fefa28d55003d75f2d562cd5323"}
Oct 30 21:53:41.312999 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:53:41.312944    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 21:53:42.110227 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:42.107563    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-bj7lq" event={"ID":"99a25eed-4eb9-4f87-8520-139ea86cf7ef","Type":"ContainerStarted","Data":"6b3549bfb58f1b7d4d5e37a0d8c0adde45737ee2f5ef80374bf6e21bc1f7917f"}
Oct 30 21:53:42.110227 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:42.109830    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-bj7lq"
Oct 30 21:53:42.110227 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:42.109935    4473 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-bj7lq container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused" start-of-body=
Oct 30 21:53:42.110227 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:42.109992    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-bj7lq" podUID="99a25eed-4eb9-4f87-8520-139ea86cf7ef" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused"
Oct 30 21:53:42.114525 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:42.114354    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs" event={"ID":"b599ef14-1c28-4c77-adbc-cb007513d00d","Type":"ContainerStarted","Data":"2ef7107236a26f517d1c9227ca3cef90d925db84a9690b4bedf8261c4e30d4a7"}
Oct 30 21:53:42.114525 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:42.114423    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs" event={"ID":"b599ef14-1c28-4c77-adbc-cb007513d00d","Type":"ContainerStarted","Data":"d79af5f199cdb90e9a9ac3527cda6fc5e7f86fbfcf9cf99bdfa0b9794fe32a70"}
Oct 30 21:53:42.114525 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:42.114462    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs" event={"ID":"b599ef14-1c28-4c77-adbc-cb007513d00d","Type":"ContainerStarted","Data":"acde812ec2c39a9320fc8262915bed2d274c55919a935949408d3cf3286b128a"}
Oct 30 21:53:42.229276 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:42.229138    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-bj7lq" podStartSLOduration=1.490175526 podStartE2EDuration="19.229117631s" totalImagesPullingTime="17.738942105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:42.228588096 +0000 UTC m=+40.464797440" watchObservedRunningTime="2026-10-30 21:53:42.229117631 +0000 UTC m=+40.465326978"
Oct 30 21:53:43.122124 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:43.122073    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs" event={"ID":"b599ef14-1c28-4c77-adbc-cb007513d00d","Type":"ContainerStarted","Data":"83bd75be82f17e4442a0b8a5617db53c1be92e760cc78be8ebcceb5a5925e286"}
Oct 30 21:53:43.133213 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:43.133184    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-bj7lq"
Oct 30 21:53:44.129345 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:44.129251    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-7l7hw" event={"ID":"d494e8a5-5577-4ce2-af1f-cec218f37b50","Type":"ContainerStarted","Data":"2c6fd36a3e7428e49a4ec3a5f85e1c1bb3e10eb3055320d53d2b36009f8fdf80"}
Oct 30 21:53:44.129823 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:44.129546    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-7l7hw"
Oct 30 21:53:44.133976 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:44.133930    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w" event={"ID":"3530dd2d-997b-48ca-9ffa-bfd102260075","Type":"ContainerStarted","Data":"6d961d01faba1f7c9856e84b1dc15be7974fa128cf667c8a2e257e0965958aa4"}
Oct 30 21:53:44.136397 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:44.136373    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-7l7hw"
Oct 30 21:53:44.137578 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:44.137552    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs" event={"ID":"b599ef14-1c28-4c77-adbc-cb007513d00d","Type":"ContainerStarted","Data":"ca6daabd4ee2d1d0e89d757f8bce15b58c7af5c991c0fca1ed2cf279c70a447b"}
Oct 30 21:53:44.137691 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:44.137619    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs" event={"ID":"b599ef14-1c28-4c77-adbc-cb007513d00d","Type":"ContainerStarted","Data":"7f29142461b5ae6b4df668bedfadbc910700f383e10eb5b7bb3f15c886fffe9d"}
Oct 30 21:53:44.137913 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:44.137893    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:44.163125 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:44.163089    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:44.163282 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:44.163158    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:44.168606 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:44.168576    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:44.246597 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:44.246539    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-7l7hw" podStartSLOduration=2.635891193 podStartE2EDuration="5.246510889s" totalImagesPullingTime="2.610619696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:44.246353004 +0000 UTC m=+42.482562350" watchObservedRunningTime="2026-10-30 21:53:44.246510889 +0000 UTC m=+42.482720238"
Oct 30 21:53:44.311345 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:44.311306    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:44.370418 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:44.370350    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w" podStartSLOduration=1.5562507 podStartE2EDuration="4.370330839s" totalImagesPullingTime="2.814080139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:44.370233881 +0000 UTC m=+42.606443227" watchObservedRunningTime="2026-10-30 21:53:44.370330839 +0000 UTC m=+42.606540182"
Oct 30 21:53:44.620106 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:44.620042    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs" podStartSLOduration=3.615678205 podStartE2EDuration="7.620022793s" totalImagesPullingTime="4.004344588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:44.618558788 +0000 UTC m=+42.854768136" watchObservedRunningTime="2026-10-30 21:53:44.620022793 +0000 UTC m=+42.856232140"
Oct 30 21:53:45.146231 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:45.146203    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:53:45.296835 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:45.296807    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84ff54fd5b-hzdg8"]
Oct 30 21:53:46.003636 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:46.003595    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:46.204999 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:46.204939    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:46.372176 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:46.372095    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:47.115041 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.115007    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:47.279783 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.279747    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:47.454929 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.454890    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f54556489-nvh6s"]
Oct 30 21:53:47.499079 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.499051    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.559387 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.559353    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f54556489-nvh6s"]
Oct 30 21:53:47.603012 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.602971    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg5q2\" (UniqueName: \"kubernetes.io/projected/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-kube-api-access-cg5q2\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.603193 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.603037    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-config\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.603193 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.603093    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-oauth-serving-cert\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.603320 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.603208    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-oauth-config\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.603320 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.603239    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-serving-cert\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.603432 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.603334    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-service-ca\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.603432 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.603389    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-tmp\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.603530 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.603493    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-trusted-ca-bundle\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.704748 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.704714    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-config\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.704976 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.704756    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-oauth-serving-cert\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.704976 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.704785    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-oauth-config\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.704976 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.704807    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-serving-cert\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.704976 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.704872    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-service-ca\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.704976 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.704911    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-tmp\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.704976 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.704939    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-trusted-ca-bundle\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.705321 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.705000    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg5q2\" (UniqueName: \"kubernetes.io/projected/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-kube-api-access-cg5q2\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.705382 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.705327    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-tmp\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.705594 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.705570    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-config\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.705736 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.705669    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-service-ca\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.705917 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.705887    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-trusted-ca-bundle\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.706190 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.706166    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-oauth-serving-cert\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.708265 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.708239    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-oauth-config\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.716425 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.716404    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-serving-cert\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.764562 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.764529    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg5q2\" (UniqueName: \"kubernetes.io/projected/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-kube-api-access-cg5q2\") pod \"console-5f54556489-nvh6s\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") " pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:47.834297 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:47.834258    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:48.234730 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:48.234696    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f54556489-nvh6s"]
Oct 30 21:53:48.238963 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:53:48.238920    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9cfeb85_8d16_4e54_b0ea_36d1975a40fc.slice/crio-78368adf74fab9e7e8ab126a6e00a1cf2d863d41ca540b0a9acda4ff5852f686 WatchSource:0}: Error finding container 78368adf74fab9e7e8ab126a6e00a1cf2d863d41ca540b0a9acda4ff5852f686: Status 404 returned error can't find the container with id 78368adf74fab9e7e8ab126a6e00a1cf2d863d41ca540b0a9acda4ff5852f686
Oct 30 21:53:49.160230 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:49.160188    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f54556489-nvh6s" event={"ID":"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc","Type":"ContainerStarted","Data":"e3d3efec8c8614d7664aff630bf96afa31f7e324df8fccb80c6512b4d9f03b06"}
Oct 30 21:53:49.160805 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:49.160251    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f54556489-nvh6s" event={"ID":"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc","Type":"ContainerStarted","Data":"78368adf74fab9e7e8ab126a6e00a1cf2d863d41ca540b0a9acda4ff5852f686"}
Oct 30 21:53:49.281632 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:49.281578    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5f54556489-nvh6s" podStartSLOduration=2.2815579440000002 podStartE2EDuration="2.281557944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:49.279608469 +0000 UTC m=+47.515817815" watchObservedRunningTime="2026-10-30 21:53:49.281557944 +0000 UTC m=+47.517767291"
Oct 30 21:53:50.149972 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:50.149919    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c57bb4c94-x2drs"
Oct 30 21:53:50.233315 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:50.233281    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:50.432041 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:50.431928    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:51.574335 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:51.574302    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:51.784249 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:51.784215    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:52.314676 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:52.314640    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:52.473557 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:52.473519    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:53.761635 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:53.761607    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:53.946606 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:53.946568    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:57.835344 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:57.835310    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:57.835761 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:57.835361    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:57.840123 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:57.840098    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:58.195775 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:58.195743    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:53:58.368569 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:53:58.368535    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-674b886b45-85zrp"]
Oct 30 21:54:00.544517 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:00.544486    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:54:00.545017 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:00.544545    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:54:02.173588 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:02.173554    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:02.373092 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:02.373060    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:04.866485 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:04.866451    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:05.022270 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:05.022235    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:10.322768 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.322737    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-84ff54fd5b-hzdg8" podUID="5c869caa-0bd6-4211-824a-ddd3c8e57a95" containerName="console" containerID="cri-o://46dfbcc342f2ce832fd4219d0cac94c3a9644a82f7ae15b718e4dd00a1a82c83" gracePeriod=15
Oct 30 21:54:10.594928 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.594856    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:54:10.601437 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.601414    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c869caa-0bd6-4211-824a-ddd3c8e57a95-kube-api-access-d7mll\" (UniqueName: \"kubernetes.io/projected/5c869caa-0bd6-4211-824a-ddd3c8e57a95-kube-api-access-d7mll\") pod \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") "
Oct 30 21:54:10.601523 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.601449    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-config\" (UniqueName: \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-config\") pod \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") "
Oct 30 21:54:10.601523 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.601486    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-serving-cert\") pod \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") "
Oct 30 21:54:10.601616 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.601593    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-oauth-config\") pod \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") "
Oct 30 21:54:10.601654 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.601628    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c869caa-0bd6-4211-824a-ddd3c8e57a95-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c869caa-0bd6-4211-824a-ddd3c8e57a95-tmp\") pod \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") "
Oct 30 21:54:10.601704 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.601664    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-oauth-serving-cert\") pod \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") "
Oct 30 21:54:10.601756 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.601715    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-service-ca\") pod \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\" (UID: \"5c869caa-0bd6-4211-824a-ddd3c8e57a95\") "
Oct 30 21:54:10.602017 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.601884    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-config" pod "5c869caa-0bd6-4211-824a-ddd3c8e57a95" (UID: "5c869caa-0bd6-4211-824a-ddd3c8e57a95"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:10.602017 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.601929    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c869caa-0bd6-4211-824a-ddd3c8e57a95-tmp" pod "5c869caa-0bd6-4211-824a-ddd3c8e57a95" (UID: "5c869caa-0bd6-4211-824a-ddd3c8e57a95"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:10.602165 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.602105    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-oauth-serving-cert" pod "5c869caa-0bd6-4211-824a-ddd3c8e57a95" (UID: "5c869caa-0bd6-4211-824a-ddd3c8e57a95"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:10.602165 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.602125    4473 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c869caa-0bd6-4211-824a-ddd3c8e57a95-tmp\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:10.602266 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.602217    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-service-ca" pod "5c869caa-0bd6-4211-824a-ddd3c8e57a95" (UID: "5c869caa-0bd6-4211-824a-ddd3c8e57a95"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:10.602304 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.602224    4473 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:10.603816 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.603796    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-serving-cert" pod "5c869caa-0bd6-4211-824a-ddd3c8e57a95" (UID: "5c869caa-0bd6-4211-824a-ddd3c8e57a95"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:10.604250 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.604230    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c869caa-0bd6-4211-824a-ddd3c8e57a95-kube-api-access-d7mll" pod "5c869caa-0bd6-4211-824a-ddd3c8e57a95" (UID: "5c869caa-0bd6-4211-824a-ddd3c8e57a95"). InnerVolumeSpecName "kube-api-access-d7mll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:10.604326 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.604240    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-oauth-config" pod "5c869caa-0bd6-4211-824a-ddd3c8e57a95" (UID: "5c869caa-0bd6-4211-824a-ddd3c8e57a95"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:10.703255 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.703222    4473 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-serving-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:10.703255 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.703248    4473 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c869caa-0bd6-4211-824a-ddd3c8e57a95-console-oauth-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:10.703255 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.703259    4473 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-oauth-serving-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:10.703485 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.703269    4473 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c869caa-0bd6-4211-824a-ddd3c8e57a95-service-ca\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:10.703485 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:10.703278    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7mll\" (UniqueName: \"kubernetes.io/projected/5c869caa-0bd6-4211-824a-ddd3c8e57a95-kube-api-access-d7mll\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:11.238294 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:11.238258    4473 generic.go:356] "Generic (PLEG): container finished" podID="5c869caa-0bd6-4211-824a-ddd3c8e57a95" containerID="46dfbcc342f2ce832fd4219d0cac94c3a9644a82f7ae15b718e4dd00a1a82c83" exitCode=0
Oct 30 21:54:11.238528 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:11.238313    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84ff54fd5b-hzdg8"
Oct 30 21:54:11.238528 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:11.238328    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84ff54fd5b-hzdg8" event={"ID":"5c869caa-0bd6-4211-824a-ddd3c8e57a95","Type":"ContainerDied","Data":"46dfbcc342f2ce832fd4219d0cac94c3a9644a82f7ae15b718e4dd00a1a82c83"}
Oct 30 21:54:11.238528 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:11.238365    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84ff54fd5b-hzdg8" event={"ID":"5c869caa-0bd6-4211-824a-ddd3c8e57a95","Type":"ContainerDied","Data":"e5c1a485cb9aadf8a79e53feac2b77a41427bd2b411c924356126f06318a695a"}
Oct 30 21:54:11.238528 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:11.238382    4473 scope.go:122] "RemoveContainer" containerID="46dfbcc342f2ce832fd4219d0cac94c3a9644a82f7ae15b718e4dd00a1a82c83"
Oct 30 21:54:11.256050 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:11.256032    4473 scope.go:122] "RemoveContainer" containerID="46dfbcc342f2ce832fd4219d0cac94c3a9644a82f7ae15b718e4dd00a1a82c83"
Oct 30 21:54:11.256343 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:54:11.256321    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46dfbcc342f2ce832fd4219d0cac94c3a9644a82f7ae15b718e4dd00a1a82c83\": container with ID starting with 46dfbcc342f2ce832fd4219d0cac94c3a9644a82f7ae15b718e4dd00a1a82c83 not found: ID does not exist" containerID="46dfbcc342f2ce832fd4219d0cac94c3a9644a82f7ae15b718e4dd00a1a82c83"
Oct 30 21:54:11.256418 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:11.256357    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46dfbcc342f2ce832fd4219d0cac94c3a9644a82f7ae15b718e4dd00a1a82c83"} err="failed to get container status \"46dfbcc342f2ce832fd4219d0cac94c3a9644a82f7ae15b718e4dd00a1a82c83\": rpc error: code = NotFound desc = could not find container \"46dfbcc342f2ce832fd4219d0cac94c3a9644a82f7ae15b718e4dd00a1a82c83\": container with ID starting with 46dfbcc342f2ce832fd4219d0cac94c3a9644a82f7ae15b718e4dd00a1a82c83 not found: ID does not exist"
Oct 30 21:54:11.304160 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:54:11.304114    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 21:54:11.412386 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:11.412358    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84ff54fd5b-hzdg8"]
Oct 30 21:54:11.462742 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:11.462714    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84ff54fd5b-hzdg8"]
Oct 30 21:54:11.885840 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:11.885810    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c869caa-0bd6-4211-824a-ddd3c8e57a95" path="/var/lib/kubelet/pods/5c869caa-0bd6-4211-824a-ddd3c8e57a95/volumes"
Oct 30 21:54:13.737191 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:13.737162    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:13.930167 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:13.930121    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:17.283860 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:17.283821    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:17.482333 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:17.482297    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:20.549911 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:20.549881    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:54:20.553766 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:20.553745    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-9b5c6b74c-5w64w"
Oct 30 21:54:23.389418 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.389380    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-674b886b45-85zrp" podUID="5a59ce7d-37c9-451f-90be-de729bab3bb3" containerName="console" containerID="cri-o://134e7069ffb9b8aa8448680cc66e1aba618f18134eac8f999ed20e9f709d2cab" gracePeriod=15
Oct 30 21:54:23.680080 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.680054    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:54:23.709421 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.709395    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-config\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-config\") pod \"5a59ce7d-37c9-451f-90be-de729bab3bb3\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") "
Oct 30 21:54:23.709574 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.709428    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a59ce7d-37c9-451f-90be-de729bab3bb3-kube-api-access-kg9hx\" (UniqueName: \"kubernetes.io/projected/5a59ce7d-37c9-451f-90be-de729bab3bb3-kube-api-access-kg9hx\") pod \"5a59ce7d-37c9-451f-90be-de729bab3bb3\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") "
Oct 30 21:54:23.709574 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.709482    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-service-ca\") pod \"5a59ce7d-37c9-451f-90be-de729bab3bb3\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") "
Oct 30 21:54:23.709574 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.709522    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-serving-cert\") pod \"5a59ce7d-37c9-451f-90be-de729bab3bb3\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") "
Oct 30 21:54:23.709574 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.709537    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-oauth-config\") pod \"5a59ce7d-37c9-451f-90be-de729bab3bb3\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") "
Oct 30 21:54:23.709574 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.709564    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-oauth-serving-cert\") pod \"5a59ce7d-37c9-451f-90be-de729bab3bb3\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") "
Oct 30 21:54:23.709818 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.709598    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a59ce7d-37c9-451f-90be-de729bab3bb3-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a59ce7d-37c9-451f-90be-de729bab3bb3-tmp\") pod \"5a59ce7d-37c9-451f-90be-de729bab3bb3\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") "
Oct 30 21:54:23.709818 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.709753    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-config" pod "5a59ce7d-37c9-451f-90be-de729bab3bb3" (UID: "5a59ce7d-37c9-451f-90be-de729bab3bb3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:23.709818 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.709765    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-trusted-ca-bundle\") pod \"5a59ce7d-37c9-451f-90be-de729bab3bb3\" (UID: \"5a59ce7d-37c9-451f-90be-de729bab3bb3\") "
Oct 30 21:54:23.709994 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.709974    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a59ce7d-37c9-451f-90be-de729bab3bb3-tmp" pod "5a59ce7d-37c9-451f-90be-de729bab3bb3" (UID: "5a59ce7d-37c9-451f-90be-de729bab3bb3"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:23.710066 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.710037    4473 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a59ce7d-37c9-451f-90be-de729bab3bb3-tmp\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:23.710066 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.710058    4473 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:23.710183 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.710163    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-service-ca" pod "5a59ce7d-37c9-451f-90be-de729bab3bb3" (UID: "5a59ce7d-37c9-451f-90be-de729bab3bb3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:23.710351 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.710244    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-oauth-serving-cert" pod "5a59ce7d-37c9-451f-90be-de729bab3bb3" (UID: "5a59ce7d-37c9-451f-90be-de729bab3bb3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:23.710587 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.710381    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-trusted-ca-bundle" pod "5a59ce7d-37c9-451f-90be-de729bab3bb3" (UID: "5a59ce7d-37c9-451f-90be-de729bab3bb3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:23.712153 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.712133    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a59ce7d-37c9-451f-90be-de729bab3bb3-kube-api-access-kg9hx" pod "5a59ce7d-37c9-451f-90be-de729bab3bb3" (UID: "5a59ce7d-37c9-451f-90be-de729bab3bb3"). InnerVolumeSpecName "kube-api-access-kg9hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:23.712432 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.712405    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-oauth-config" pod "5a59ce7d-37c9-451f-90be-de729bab3bb3" (UID: "5a59ce7d-37c9-451f-90be-de729bab3bb3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:23.712514 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.712430    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-serving-cert" pod "5a59ce7d-37c9-451f-90be-de729bab3bb3" (UID: "5a59ce7d-37c9-451f-90be-de729bab3bb3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:23.811212 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.811180    4473 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-service-ca\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:23.811212 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.811207    4473 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-serving-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:23.811212 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.811218    4473 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a59ce7d-37c9-451f-90be-de729bab3bb3-console-oauth-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:23.811442 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.811228    4473 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-oauth-serving-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:23.811442 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.811237    4473 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a59ce7d-37c9-451f-90be-de729bab3bb3-trusted-ca-bundle\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:23.811442 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:23.811245    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg9hx\" (UniqueName: \"kubernetes.io/projected/5a59ce7d-37c9-451f-90be-de729bab3bb3-kube-api-access-kg9hx\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:24.286121 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:24.286087    4473 generic.go:356] "Generic (PLEG): container finished" podID="5a59ce7d-37c9-451f-90be-de729bab3bb3" containerID="134e7069ffb9b8aa8448680cc66e1aba618f18134eac8f999ed20e9f709d2cab" exitCode=0
Oct 30 21:54:24.286321 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:24.286130    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-674b886b45-85zrp"
Oct 30 21:54:24.286321 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:24.286170    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-674b886b45-85zrp" event={"ID":"5a59ce7d-37c9-451f-90be-de729bab3bb3","Type":"ContainerDied","Data":"134e7069ffb9b8aa8448680cc66e1aba618f18134eac8f999ed20e9f709d2cab"}
Oct 30 21:54:24.286321 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:24.286233    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-674b886b45-85zrp" event={"ID":"5a59ce7d-37c9-451f-90be-de729bab3bb3","Type":"ContainerDied","Data":"0ac3ac91d59a8b935af12fce7dc3c0c8ca07075ff8ec32ae467aee10abf6b14d"}
Oct 30 21:54:24.286321 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:24.286263    4473 scope.go:122] "RemoveContainer" containerID="134e7069ffb9b8aa8448680cc66e1aba618f18134eac8f999ed20e9f709d2cab"
Oct 30 21:54:24.294500 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:24.294480    4473 scope.go:122] "RemoveContainer" containerID="134e7069ffb9b8aa8448680cc66e1aba618f18134eac8f999ed20e9f709d2cab"
Oct 30 21:54:24.294798 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:54:24.294775    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134e7069ffb9b8aa8448680cc66e1aba618f18134eac8f999ed20e9f709d2cab\": container with ID starting with 134e7069ffb9b8aa8448680cc66e1aba618f18134eac8f999ed20e9f709d2cab not found: ID does not exist" containerID="134e7069ffb9b8aa8448680cc66e1aba618f18134eac8f999ed20e9f709d2cab"
Oct 30 21:54:24.294855 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:24.294812    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134e7069ffb9b8aa8448680cc66e1aba618f18134eac8f999ed20e9f709d2cab"} err="failed to get container status \"134e7069ffb9b8aa8448680cc66e1aba618f18134eac8f999ed20e9f709d2cab\": rpc error: code = NotFound desc = could not find container \"134e7069ffb9b8aa8448680cc66e1aba618f18134eac8f999ed20e9f709d2cab\": container with ID starting with 134e7069ffb9b8aa8448680cc66e1aba618f18134eac8f999ed20e9f709d2cab not found: ID does not exist"
Oct 30 21:54:24.451536 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:24.451505    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-674b886b45-85zrp"]
Oct 30 21:54:24.501353 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:24.501326    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-674b886b45-85zrp"]
Oct 30 21:54:25.885991 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:25.885937    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a59ce7d-37c9-451f-90be-de729bab3bb3" path="/var/lib/kubelet/pods/5a59ce7d-37c9-451f-90be-de729bab3bb3/volumes"
Oct 30 21:54:28.396487 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:28.396452    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:28.645641 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:28.645608    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:30.311070 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:30.311035    4473 generic.go:356] "Generic (PLEG): container finished" podID="8faee8d2-4d27-4709-af52-4b5ef32592f6" containerID="41589b06dc449203e8d4f4eb7548f30fd0d475474b4fe41974f5835411426ad5" exitCode=0
Oct 30 21:54:30.311491 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:30.311084    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w68zw-mzlkv" event={"ID":"8faee8d2-4d27-4709-af52-4b5ef32592f6","Type":"ContainerDied","Data":"41589b06dc449203e8d4f4eb7548f30fd0d475474b4fe41974f5835411426ad5"}
Oct 30 21:54:31.447040 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:31.447012    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:54:31.575549 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:31.575471    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-trusted-ca-bundle\") pod \"8faee8d2-4d27-4709-af52-4b5ef32592f6\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") "
Oct 30 21:54:31.575687 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:31.575571    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-service-ca-bundle\") pod \"8faee8d2-4d27-4709-af52-4b5ef32592f6\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") "
Oct 30 21:54:31.575687 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:31.575609    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8faee8d2-4d27-4709-af52-4b5ef32592f6-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8faee8d2-4d27-4709-af52-4b5ef32592f6-archives-path\") pod \"8faee8d2-4d27-4709-af52-4b5ef32592f6\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") "
Oct 30 21:54:31.575687 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:31.575638    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8faee8d2-4d27-4709-af52-4b5ef32592f6-kube-api-access-m744c\" (UniqueName: \"kubernetes.io/projected/8faee8d2-4d27-4709-af52-4b5ef32592f6-kube-api-access-m744c\") pod \"8faee8d2-4d27-4709-af52-4b5ef32592f6\" (UID: \"8faee8d2-4d27-4709-af52-4b5ef32592f6\") "
Oct 30 21:54:31.575938 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:31.575914    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-trusted-ca-bundle" pod "8faee8d2-4d27-4709-af52-4b5ef32592f6" (UID: "8faee8d2-4d27-4709-af52-4b5ef32592f6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:31.576014 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:31.575932    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-service-ca-bundle" pod "8faee8d2-4d27-4709-af52-4b5ef32592f6" (UID: "8faee8d2-4d27-4709-af52-4b5ef32592f6"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:31.576014 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:31.575913    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8faee8d2-4d27-4709-af52-4b5ef32592f6-archives-path" pod "8faee8d2-4d27-4709-af52-4b5ef32592f6" (UID: "8faee8d2-4d27-4709-af52-4b5ef32592f6"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:31.578297 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:31.578263    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8faee8d2-4d27-4709-af52-4b5ef32592f6-kube-api-access-m744c" pod "8faee8d2-4d27-4709-af52-4b5ef32592f6" (UID: "8faee8d2-4d27-4709-af52-4b5ef32592f6"). InnerVolumeSpecName "kube-api-access-m744c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:31.677123 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:31.677086    4473 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-service-ca-bundle\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.677123 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:31.677118    4473 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8faee8d2-4d27-4709-af52-4b5ef32592f6-archives-path\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.677123 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:31.677131    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m744c\" (UniqueName: \"kubernetes.io/projected/8faee8d2-4d27-4709-af52-4b5ef32592f6-kube-api-access-m744c\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.677362 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:31.677145    4473 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8faee8d2-4d27-4709-af52-4b5ef32592f6-trusted-ca-bundle\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:32.206163 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:32.206133    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:32.321191 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:32.321163    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w68zw-mzlkv"
Oct 30 21:54:32.321191 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:32.321181    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w68zw-mzlkv" event={"ID":"8faee8d2-4d27-4709-af52-4b5ef32592f6","Type":"ContainerDied","Data":"6d3fe2f59da01374bc7947e14cfb67ac02039b9d7a0ba6a6036cee13bb2d3b6f"}
Oct 30 21:54:32.321400 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:32.321221    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d3fe2f59da01374bc7947e14cfb67ac02039b9d7a0ba6a6036cee13bb2d3b6f"
Oct 30 21:54:32.445868 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:32.445834    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:33.487333 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:33.487299    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:33.686200 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:33.686164    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:41.303653 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:54:41.303615    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 21:54:51.942631 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:51.942595    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:53.680511 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:53.680469    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:59.024495 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:59.024460    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:59.297558 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:54:59.297477    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:10.935944 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.935914    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-758bbd9745-tpl5z"]
Oct 30 21:55:10.936380 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.936258    4473 cpu_manager.go:414] "containerMap: removing container" podUID="8faee8d2-4d27-4709-af52-4b5ef32592f6" containerName="insights-gathering"
Oct 30 21:55:10.936380 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.936271    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8faee8d2-4d27-4709-af52-4b5ef32592f6" containerName="insights-gathering"
Oct 30 21:55:10.936380 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.936287    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5a59ce7d-37c9-451f-90be-de729bab3bb3" containerName="console"
Oct 30 21:55:10.936380 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.936295    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a59ce7d-37c9-451f-90be-de729bab3bb3" containerName="console"
Oct 30 21:55:10.936380 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.936310    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5c869caa-0bd6-4211-824a-ddd3c8e57a95" containerName="console"
Oct 30 21:55:10.936380 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.936314    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c869caa-0bd6-4211-824a-ddd3c8e57a95" containerName="console"
Oct 30 21:55:10.936380 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.936361    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8faee8d2-4d27-4709-af52-4b5ef32592f6" containerName="insights-gathering"
Oct 30 21:55:10.936380 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.936370    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a59ce7d-37c9-451f-90be-de729bab3bb3" containerName="console"
Oct 30 21:55:10.936380 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.936381    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c869caa-0bd6-4211-824a-ddd3c8e57a95" containerName="console"
Oct 30 21:55:10.938065 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.938049    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:10.980228 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.980198    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-serving-cert\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:10.980228 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.980233    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-oauth-config\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:10.980425 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.980255    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-service-ca\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:10.980425 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.980276    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-tmp\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:10.980425 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.980416    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4glqb\" (UniqueName: \"kubernetes.io/projected/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-kube-api-access-4glqb\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:10.980522 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.980447    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-config\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:10.980522 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.980470    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-trusted-ca-bundle\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:10.980581 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:10.980527    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-oauth-serving-cert\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.042165 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.042130    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-758bbd9745-tpl5z"]
Oct 30 21:55:11.080971 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.080912    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-service-ca\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.080971 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.080976    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-tmp\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.081191 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.081029    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4glqb\" (UniqueName: \"kubernetes.io/projected/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-kube-api-access-4glqb\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.081191 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.081059    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-config\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.081191 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.081086    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-trusted-ca-bundle\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.081191 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.081109    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-oauth-serving-cert\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.081403 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.081244    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-serving-cert\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.081403 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.081298    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-oauth-config\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.081512 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.081496    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-tmp\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.081793 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.081768    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-oauth-serving-cert\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.081898 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.081801    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-service-ca\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.081975 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.081922    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-config\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.082244 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.082215    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-trusted-ca-bundle\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.083911 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.083884    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-serving-cert\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.084039 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.083927    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-oauth-config\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.138487 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.138450    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4glqb\" (UniqueName: \"kubernetes.io/projected/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-kube-api-access-4glqb\") pod \"console-758bbd9745-tpl5z\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") " pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.230104 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.230024    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:11.248611 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.248589    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:11.309931 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:55:11.309867    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 21:55:11.391753 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.391702    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:11.607237 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.607154    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:11.620284 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.620255    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-758bbd9745-tpl5z"]
Oct 30 21:55:11.651374 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:55:11.651345    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d2c5c3_2a3e_404b_936c_6727eaa2ddd1.slice/crio-e908bac4cbc14d6dd7afd690974478ca270bb7e198b8fe519cf34d2489b35502 WatchSource:0}: Error finding container e908bac4cbc14d6dd7afd690974478ca270bb7e198b8fe519cf34d2489b35502: Status 404 returned error can't find the container with id e908bac4cbc14d6dd7afd690974478ca270bb7e198b8fe519cf34d2489b35502
Oct 30 21:55:11.774238 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:11.774207    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:12.464120 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:12.464032    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758bbd9745-tpl5z" event={"ID":"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1","Type":"ContainerStarted","Data":"ae82d4422a5145f69e29a844248edd41b182d3dc158393cfc0a56cb798213dca"}
Oct 30 21:55:12.464120 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:12.464085    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758bbd9745-tpl5z" event={"ID":"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1","Type":"ContainerStarted","Data":"e908bac4cbc14d6dd7afd690974478ca270bb7e198b8fe519cf34d2489b35502"}
Oct 30 21:55:12.582600 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:12.582546    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-758bbd9745-tpl5z" podStartSLOduration=2.582531075 podStartE2EDuration="2.582531075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:12.582127715 +0000 UTC m=+130.818337073" watchObservedRunningTime="2026-10-30 21:55:12.582531075 +0000 UTC m=+130.818740416"
Oct 30 21:55:21.248712 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:21.248673    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:21.249190 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:21.248753    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:21.253499 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:21.253474    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:21.500306 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:21.500229    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:55:21.646579 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:21.646543    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f54556489-nvh6s"]
Oct 30 21:55:22.356297 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:22.356265    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:41.302722 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:55:41.302679    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 21:55:46.665856 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.665797    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5f54556489-nvh6s" podUID="c9cfeb85-8d16-4e54-b0ea-36d1975a40fc" containerName="console" containerID="cri-o://e3d3efec8c8614d7664aff630bf96afa31f7e324df8fccb80c6512b4d9f03b06" gracePeriod=15
Oct 30 21:55:46.916886 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.916835    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:55:46.951344 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.951315    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-kube-api-access-cg5q2\" (UniqueName: \"kubernetes.io/projected/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-kube-api-access-cg5q2\") pod \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") "
Oct 30 21:55:46.951467 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.951355    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-config\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-config\") pod \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") "
Oct 30 21:55:46.951467 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.951381    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-service-ca\") pod \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") "
Oct 30 21:55:46.951467 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.951452    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-tmp\") pod \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") "
Oct 30 21:55:46.951641 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.951490    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-trusted-ca-bundle\") pod \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") "
Oct 30 21:55:46.951641 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.951532    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-oauth-config\") pod \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") "
Oct 30 21:55:46.951641 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.951561    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-serving-cert\") pod \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") "
Oct 30 21:55:46.951641 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.951597    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-oauth-serving-cert\") pod \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\" (UID: \"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc\") "
Oct 30 21:55:46.951822 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.951804    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-service-ca" pod "c9cfeb85-8d16-4e54-b0ea-36d1975a40fc" (UID: "c9cfeb85-8d16-4e54-b0ea-36d1975a40fc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:46.951874 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.951821    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-tmp" pod "c9cfeb85-8d16-4e54-b0ea-36d1975a40fc" (UID: "c9cfeb85-8d16-4e54-b0ea-36d1975a40fc"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:46.951874 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.951845    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-config" pod "c9cfeb85-8d16-4e54-b0ea-36d1975a40fc" (UID: "c9cfeb85-8d16-4e54-b0ea-36d1975a40fc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:46.952200 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.952174    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-trusted-ca-bundle" pod "c9cfeb85-8d16-4e54-b0ea-36d1975a40fc" (UID: "c9cfeb85-8d16-4e54-b0ea-36d1975a40fc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:46.952200 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.952184    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-oauth-serving-cert" pod "c9cfeb85-8d16-4e54-b0ea-36d1975a40fc" (UID: "c9cfeb85-8d16-4e54-b0ea-36d1975a40fc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:46.953611 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.953587    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-kube-api-access-cg5q2" pod "c9cfeb85-8d16-4e54-b0ea-36d1975a40fc" (UID: "c9cfeb85-8d16-4e54-b0ea-36d1975a40fc"). InnerVolumeSpecName "kube-api-access-cg5q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:46.953762 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.953744    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-serving-cert" pod "c9cfeb85-8d16-4e54-b0ea-36d1975a40fc" (UID: "c9cfeb85-8d16-4e54-b0ea-36d1975a40fc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:46.953895 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:46.953871    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-oauth-config" pod "c9cfeb85-8d16-4e54-b0ea-36d1975a40fc" (UID: "c9cfeb85-8d16-4e54-b0ea-36d1975a40fc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:47.052623 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.052590    4473 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-oauth-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:47.052623 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.052620    4473 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-serving-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:47.052623 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.052631    4473 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-oauth-serving-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:47.052841 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.052639    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg5q2\" (UniqueName: \"kubernetes.io/projected/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-kube-api-access-cg5q2\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:47.052841 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.052652    4473 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-console-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:47.052841 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.052662    4473 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-service-ca\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:47.052841 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.052671    4473 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-tmp\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:47.052841 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.052681    4473 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc-trusted-ca-bundle\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:47.583011 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.582975    4473 generic.go:356] "Generic (PLEG): container finished" podID="c9cfeb85-8d16-4e54-b0ea-36d1975a40fc" containerID="e3d3efec8c8614d7664aff630bf96afa31f7e324df8fccb80c6512b4d9f03b06" exitCode=0
Oct 30 21:55:47.583011 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.583016    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f54556489-nvh6s"
Oct 30 21:55:47.583011 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.583019    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f54556489-nvh6s" event={"ID":"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc","Type":"ContainerDied","Data":"e3d3efec8c8614d7664aff630bf96afa31f7e324df8fccb80c6512b4d9f03b06"}
Oct 30 21:55:47.583296 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.583067    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f54556489-nvh6s" event={"ID":"c9cfeb85-8d16-4e54-b0ea-36d1975a40fc","Type":"ContainerDied","Data":"78368adf74fab9e7e8ab126a6e00a1cf2d863d41ca540b0a9acda4ff5852f686"}
Oct 30 21:55:47.583296 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.583091    4473 scope.go:122] "RemoveContainer" containerID="e3d3efec8c8614d7664aff630bf96afa31f7e324df8fccb80c6512b4d9f03b06"
Oct 30 21:55:47.591374 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.591351    4473 scope.go:122] "RemoveContainer" containerID="e3d3efec8c8614d7664aff630bf96afa31f7e324df8fccb80c6512b4d9f03b06"
Oct 30 21:55:47.591655 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:55:47.591634    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d3efec8c8614d7664aff630bf96afa31f7e324df8fccb80c6512b4d9f03b06\": container with ID starting with e3d3efec8c8614d7664aff630bf96afa31f7e324df8fccb80c6512b4d9f03b06 not found: ID does not exist" containerID="e3d3efec8c8614d7664aff630bf96afa31f7e324df8fccb80c6512b4d9f03b06"
Oct 30 21:55:47.591700 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.591664    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d3efec8c8614d7664aff630bf96afa31f7e324df8fccb80c6512b4d9f03b06"} err="failed to get container status \"e3d3efec8c8614d7664aff630bf96afa31f7e324df8fccb80c6512b4d9f03b06\": rpc error: code = NotFound desc = could not find container \"e3d3efec8c8614d7664aff630bf96afa31f7e324df8fccb80c6512b4d9f03b06\": container with ID starting with e3d3efec8c8614d7664aff630bf96afa31f7e324df8fccb80c6512b4d9f03b06 not found: ID does not exist"
Oct 30 21:55:47.748648 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.748602    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f54556489-nvh6s"]
Oct 30 21:55:47.799320 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.799284    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f54556489-nvh6s"]
Oct 30 21:55:47.886827 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:47.886758    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9cfeb85-8d16-4e54-b0ea-36d1975a40fc" path="/var/lib/kubelet/pods/c9cfeb85-8d16-4e54-b0ea-36d1975a40fc/volumes"
Oct 30 21:55:51.341202 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:51.341163    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:52.192606 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:55:52.192573    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:04.623058 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:04.623023    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:11.303462 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:56:11.303428    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 21:56:16.915139 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:16.915101    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:17.097185 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:17.097153    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:27.551008 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.550975    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fc844489c-7k9tt"]
Oct 30 21:56:27.551407 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.551243    4473 cpu_manager.go:414] "containerMap: removing container" podUID="c9cfeb85-8d16-4e54-b0ea-36d1975a40fc" containerName="console"
Oct 30 21:56:27.551407 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.551253    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9cfeb85-8d16-4e54-b0ea-36d1975a40fc" containerName="console"
Oct 30 21:56:27.551407 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.551320    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9cfeb85-8d16-4e54-b0ea-36d1975a40fc" containerName="console"
Oct 30 21:56:27.553233 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.553219    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.613533 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.613504    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fc844489c-7k9tt"]
Oct 30 21:56:27.661547 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.661519    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e69d3e-5768-4c46-8d25-d934422c8d4d-service-ca\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.661701 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.661559    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxgcv\" (UniqueName: \"kubernetes.io/projected/e2e69d3e-5768-4c46-8d25-d934422c8d4d-kube-api-access-sxgcv\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.661701 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.661577    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2e69d3e-5768-4c46-8d25-d934422c8d4d-trusted-ca-bundle\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.661701 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.661634    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2e69d3e-5768-4c46-8d25-d934422c8d4d-tmp\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.661810 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.661733    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2e69d3e-5768-4c46-8d25-d934422c8d4d-console-oauth-config\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.661810 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.661766    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2e69d3e-5768-4c46-8d25-d934422c8d4d-console-config\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.661810 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.661788    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2e69d3e-5768-4c46-8d25-d934422c8d4d-oauth-serving-cert\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.661904 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.661867    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2e69d3e-5768-4c46-8d25-d934422c8d4d-console-serving-cert\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.762433 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.762382    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2e69d3e-5768-4c46-8d25-d934422c8d4d-console-config\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.762433 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.762438    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2e69d3e-5768-4c46-8d25-d934422c8d4d-oauth-serving-cert\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.762716 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.762467    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2e69d3e-5768-4c46-8d25-d934422c8d4d-console-serving-cert\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.762716 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.762487    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e69d3e-5768-4c46-8d25-d934422c8d4d-service-ca\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.762716 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.762511    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxgcv\" (UniqueName: \"kubernetes.io/projected/e2e69d3e-5768-4c46-8d25-d934422c8d4d-kube-api-access-sxgcv\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.762716 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.762527    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2e69d3e-5768-4c46-8d25-d934422c8d4d-trusted-ca-bundle\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.762716 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.762544    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2e69d3e-5768-4c46-8d25-d934422c8d4d-tmp\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.762716 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.762563    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2e69d3e-5768-4c46-8d25-d934422c8d4d-console-oauth-config\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.763059 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.762990    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2e69d3e-5768-4c46-8d25-d934422c8d4d-tmp\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.763282 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.763261    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2e69d3e-5768-4c46-8d25-d934422c8d4d-console-config\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.763526 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.763415    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e69d3e-5768-4c46-8d25-d934422c8d4d-service-ca\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.763571 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.763470    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2e69d3e-5768-4c46-8d25-d934422c8d4d-oauth-serving-cert\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.763604 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.763567    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2e69d3e-5768-4c46-8d25-d934422c8d4d-trusted-ca-bundle\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.765360 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.765333    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2e69d3e-5768-4c46-8d25-d934422c8d4d-console-serving-cert\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.765454 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.765333    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2e69d3e-5768-4c46-8d25-d934422c8d4d-console-oauth-config\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.821100 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.821039    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxgcv\" (UniqueName: \"kubernetes.io/projected/e2e69d3e-5768-4c46-8d25-d934422c8d4d-kube-api-access-sxgcv\") pod \"console-6fc844489c-7k9tt\" (UID: \"e2e69d3e-5768-4c46-8d25-d934422c8d4d\") " pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:27.849774 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.849745    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:27.863742 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:27.863722    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:28.018709 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:28.018677    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:28.224357 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:28.224314    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:28.283700 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:28.283678    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fc844489c-7k9tt"]
Oct 30 21:56:28.286058 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:56:28.286031    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e69d3e_5768_4c46_8d25_d934422c8d4d.slice/crio-8644067a21fc36f9abf190472e776c3681526009e2507acd84bfbaa34069955d WatchSource:0}: Error finding container 8644067a21fc36f9abf190472e776c3681526009e2507acd84bfbaa34069955d: Status 404 returned error can't find the container with id 8644067a21fc36f9abf190472e776c3681526009e2507acd84bfbaa34069955d
Oct 30 21:56:28.374810 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:28.374775    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:28.729505 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:28.729468    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fc844489c-7k9tt" event={"ID":"e2e69d3e-5768-4c46-8d25-d934422c8d4d","Type":"ContainerStarted","Data":"568b4200488eefc77686062ca1ef64db37468d62979abee2a9d0b6cb3361b3c1"}
Oct 30 21:56:28.729896 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:28.729521    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fc844489c-7k9tt" event={"ID":"e2e69d3e-5768-4c46-8d25-d934422c8d4d","Type":"ContainerStarted","Data":"8644067a21fc36f9abf190472e776c3681526009e2507acd84bfbaa34069955d"}
Oct 30 21:56:28.847992 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:28.847915    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6fc844489c-7k9tt" podStartSLOduration=1.847899207 podStartE2EDuration="1.847899207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:28.847738931 +0000 UTC m=+207.083948277" watchObservedRunningTime="2026-10-30 21:56:28.847899207 +0000 UTC m=+207.084108553"
Oct 30 21:56:29.600502 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:29.600469    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:29.772863 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:29.772833    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:32.111344 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:32.111307    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:32.263186 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:32.263153    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:37.864682 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:37.864646    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:37.865100 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:37.864941    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:37.869510 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:37.869491    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:38.770015 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:38.769980    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fc844489c-7k9tt"
Oct 30 21:56:38.940647 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:38.940611    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-758bbd9745-tpl5z"]
Oct 30 21:56:41.303169 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:56:41.303131    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 21:56:52.548195 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:56:52.548100    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:03.960323 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:03.960283    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-758bbd9745-tpl5z" podUID="e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1" containerName="console" containerID="cri-o://ae82d4422a5145f69e29a844248edd41b182d3dc158393cfc0a56cb798213dca" gracePeriod=15
Oct 30 21:57:04.198098 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.198071    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:57:04.340541 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.340461    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-oauth-config\") pod \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") "
Oct 30 21:57:04.340541 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.340500    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-config\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-config\") pod \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") "
Oct 30 21:57:04.340541 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.340536    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-tmp\") pod \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") "
Oct 30 21:57:04.340805 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.340582    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-kube-api-access-4glqb\" (UniqueName: \"kubernetes.io/projected/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-kube-api-access-4glqb\") pod \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") "
Oct 30 21:57:04.340805 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.340619    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-trusted-ca-bundle\") pod \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") "
Oct 30 21:57:04.340805 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.340645    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-oauth-serving-cert\") pod \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") "
Oct 30 21:57:04.340805 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.340732    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-service-ca\") pod \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") "
Oct 30 21:57:04.340805 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.340784    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-serving-cert\") pod \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\" (UID: \"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1\") "
Oct 30 21:57:04.341074 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.340899    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-tmp" pod "e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1" (UID: "e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:04.341074 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.341055    4473 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-tmp\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.341171 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.341098    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-config" pod "e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1" (UID: "e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:04.341171 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.341119    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-trusted-ca-bundle" pod "e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1" (UID: "e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:04.341278 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.341210    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-service-ca" pod "e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1" (UID: "e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:04.341327 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.341285    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-oauth-serving-cert" pod "e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1" (UID: "e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:04.342856 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.342830    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-oauth-config" pod "e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1" (UID: "e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:04.342995 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.342937    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-kube-api-access-4glqb" pod "e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1" (UID: "e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1"). InnerVolumeSpecName "kube-api-access-4glqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:04.343300 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.343285    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-serving-cert" pod "e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1" (UID: "e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:04.442341 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.442309    4473 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-serving-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.442341 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.442334    4473 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-oauth-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.442341 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.442344    4473 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-console-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.442571 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.442353    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4glqb\" (UniqueName: \"kubernetes.io/projected/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-kube-api-access-4glqb\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.442571 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.442364    4473 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-trusted-ca-bundle\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.442571 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.442373    4473 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-oauth-serving-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.442571 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.442381    4473 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1-service-ca\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.856151 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.856111    4473 generic.go:356] "Generic (PLEG): container finished" podID="e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1" containerID="ae82d4422a5145f69e29a844248edd41b182d3dc158393cfc0a56cb798213dca" exitCode=0
Oct 30 21:57:04.856321 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.856160    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758bbd9745-tpl5z"
Oct 30 21:57:04.856321 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.856160    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758bbd9745-tpl5z" event={"ID":"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1","Type":"ContainerDied","Data":"ae82d4422a5145f69e29a844248edd41b182d3dc158393cfc0a56cb798213dca"}
Oct 30 21:57:04.856321 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.856220    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758bbd9745-tpl5z" event={"ID":"e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1","Type":"ContainerDied","Data":"e908bac4cbc14d6dd7afd690974478ca270bb7e198b8fe519cf34d2489b35502"}
Oct 30 21:57:04.856321 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.856242    4473 scope.go:122] "RemoveContainer" containerID="ae82d4422a5145f69e29a844248edd41b182d3dc158393cfc0a56cb798213dca"
Oct 30 21:57:04.869733 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.869712    4473 scope.go:122] "RemoveContainer" containerID="ae82d4422a5145f69e29a844248edd41b182d3dc158393cfc0a56cb798213dca"
Oct 30 21:57:04.870040 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:57:04.870019    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae82d4422a5145f69e29a844248edd41b182d3dc158393cfc0a56cb798213dca\": container with ID starting with ae82d4422a5145f69e29a844248edd41b182d3dc158393cfc0a56cb798213dca not found: ID does not exist" containerID="ae82d4422a5145f69e29a844248edd41b182d3dc158393cfc0a56cb798213dca"
Oct 30 21:57:04.870108 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:04.870050    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae82d4422a5145f69e29a844248edd41b182d3dc158393cfc0a56cb798213dca"} err="failed to get container status \"ae82d4422a5145f69e29a844248edd41b182d3dc158393cfc0a56cb798213dca\": rpc error: code = NotFound desc = could not find container \"ae82d4422a5145f69e29a844248edd41b182d3dc158393cfc0a56cb798213dca\": container with ID starting with ae82d4422a5145f69e29a844248edd41b182d3dc158393cfc0a56cb798213dca not found: ID does not exist"
Oct 30 21:57:05.021464 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:05.021428    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-758bbd9745-tpl5z"]
Oct 30 21:57:05.071939 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:05.071905    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-758bbd9745-tpl5z"]
Oct 30 21:57:05.884899 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:05.884867    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1" path="/var/lib/kubelet/pods/e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1/volumes"
Oct 30 21:57:11.303447 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:57:11.303409    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 21:57:41.303251 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:57:41.303215    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 21:57:52.885208 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:57:52.885177    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:01.874839 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:01.874813    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 21:58:01.890825 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:01.890805    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 21:58:11.303620 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:58:11.303583    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 21:58:21.092079 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:21.091944    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:21.364685 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:21.364566    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:22.488565 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.488527    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb"]
Oct 30 21:58:22.488974 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.488807    4473 cpu_manager.go:414] "containerMap: removing container" podUID="e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1" containerName="console"
Oct 30 21:58:22.488974 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.488817    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1" containerName="console"
Oct 30 21:58:22.488974 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.488875    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d2c5c3-2a3e-404b-936c-6727eaa2ddd1" containerName="console"
Oct 30 21:58:22.491870 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.491854    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb"
Oct 30 21:58:22.595516 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.595476    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 21:58:22.595516 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.595476    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"98a21a31f7a8ae0212b9c22cc643f9202bfa37-6b211c7f-dockercfg-kpxkk\""
Oct 30 21:58:22.595516 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.595491    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 21:58:22.596552 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.596533    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb"]
Oct 30 21:58:22.615193 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.615165    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdgt5\" (UniqueName: \"kubernetes.io/projected/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-kube-api-access-kdgt5\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb\" (UID: \"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb"
Oct 30 21:58:22.615349 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.615203    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-bundle\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb\" (UID: \"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb"
Oct 30 21:58:22.615349 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.615232    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-util\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb\" (UID: \"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb"
Oct 30 21:58:22.715560 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.715529    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-util\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb\" (UID: \"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb"
Oct 30 21:58:22.715735 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.715602    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdgt5\" (UniqueName: \"kubernetes.io/projected/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-kube-api-access-kdgt5\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb\" (UID: \"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb"
Oct 30 21:58:22.715735 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.715629    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-bundle\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb\" (UID: \"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb"
Oct 30 21:58:22.715936 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.715913    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-util\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb\" (UID: \"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb"
Oct 30 21:58:22.716075 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.715988    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-bundle\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb\" (UID: \"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb"
Oct 30 21:58:22.774447 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.774379    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdgt5\" (UniqueName: \"kubernetes.io/projected/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-kube-api-access-kdgt5\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb\" (UID: \"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb"
Oct 30 21:58:22.806856 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:22.806830    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb"
Oct 30 21:58:23.181887 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:23.181863    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb"]
Oct 30 21:58:23.184506 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:58:23.184479    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c957ae_e3ab_4ec0_8dc6_e2ed8f88f296.slice/crio-0ee94e853485cab8fa8903796f11f43b926b4bd347d4ec934bfaad9e1ee4a300 WatchSource:0}: Error finding container 0ee94e853485cab8fa8903796f11f43b926b4bd347d4ec934bfaad9e1ee4a300: Status 404 returned error can't find the container with id 0ee94e853485cab8fa8903796f11f43b926b4bd347d4ec934bfaad9e1ee4a300
Oct 30 21:58:23.186353 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:23.186339    4473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:58:24.123715 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:24.123683    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb" event={"ID":"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296","Type":"ContainerStarted","Data":"0ee94e853485cab8fa8903796f11f43b926b4bd347d4ec934bfaad9e1ee4a300"}
Oct 30 21:58:30.150405 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:30.150371    4473 generic.go:356] "Generic (PLEG): container finished" podID="f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296" containerID="ff1ff1b83c322671081be8a7ace58b1403822a9acc3f67a0433f66ea4bb18b89" exitCode=0
Oct 30 21:58:30.150829 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:30.150429    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb" event={"ID":"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296","Type":"ContainerDied","Data":"ff1ff1b83c322671081be8a7ace58b1403822a9acc3f67a0433f66ea4bb18b89"}
Oct 30 21:58:34.166637 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:34.166598    4473 generic.go:356] "Generic (PLEG): container finished" podID="f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296" containerID="4707e4b85f5fb46539cfe06451e81c53a043b1f118a52ecaa72136632f82ef58" exitCode=0
Oct 30 21:58:34.167049 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:34.166679    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb" event={"ID":"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296","Type":"ContainerDied","Data":"4707e4b85f5fb46539cfe06451e81c53a043b1f118a52ecaa72136632f82ef58"}
Oct 30 21:58:41.197511 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:41.197472    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb" event={"ID":"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296","Type":"ContainerStarted","Data":"718fe735e6c41777c1c95e592490a04af97175b61da1fe37e3c303a4653c9bba"}
Oct 30 21:58:41.303872 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:58:41.303831    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 21:58:41.316804 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:41.316754    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb" podStartSLOduration=3.001831964 podStartE2EDuration="19.316738925s" totalImagesPullingTime="16.314906961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:41.316082871 +0000 UTC m=+339.552292216" watchObservedRunningTime="2026-10-30 21:58:41.316738925 +0000 UTC m=+339.552948308"
Oct 30 21:58:42.203405 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:42.203367    4473 generic.go:356] "Generic (PLEG): container finished" podID="f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296" containerID="718fe735e6c41777c1c95e592490a04af97175b61da1fe37e3c303a4653c9bba" exitCode=0
Oct 30 21:58:42.203799 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:42.203461    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb" event={"ID":"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296","Type":"ContainerDied","Data":"718fe735e6c41777c1c95e592490a04af97175b61da1fe37e3c303a4653c9bba"}
Oct 30 21:58:43.379404 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:43.379374    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb"
Oct 30 21:58:43.490181 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:43.490106    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-kube-api-access-kdgt5\" (UniqueName: \"kubernetes.io/projected/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-kube-api-access-kdgt5\") pod \"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296\" (UID: \"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296\") "
Oct 30 21:58:43.490332 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:43.490184    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-bundle\") pod \"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296\" (UID: \"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296\") "
Oct 30 21:58:43.490332 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:43.490229    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-util\" (UniqueName: \"kubernetes.io/empty-dir/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-util\") pod \"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296\" (UID: \"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296\") "
Oct 30 21:58:43.490855 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:43.490827    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-bundle" pod "f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296" (UID: "f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:43.492442 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:43.492416    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-kube-api-access-kdgt5" pod "f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296" (UID: "f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296"). InnerVolumeSpecName "kube-api-access-kdgt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:43.495086 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:43.495068    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-util" pod "f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296" (UID: "f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:43.591021 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:43.590984    4473 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-util\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:43.591021 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:43.591013    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdgt5\" (UniqueName: \"kubernetes.io/projected/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-kube-api-access-kdgt5\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:43.591021 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:43.591025    4473 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296-bundle\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:44.212603 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:44.212575    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb"
Oct 30 21:58:44.212766 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:44.212574    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cmfxgb" event={"ID":"f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296","Type":"ContainerDied","Data":"0ee94e853485cab8fa8903796f11f43b926b4bd347d4ec934bfaad9e1ee4a300"}
Oct 30 21:58:44.212766 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:44.212691    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ee94e853485cab8fa8903796f11f43b926b4bd347d4ec934bfaad9e1ee4a300"
Oct 30 21:58:49.012603 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:49.012567    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:49.240381 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:49.240347    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:53.265082 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:53.265039    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:57.049800 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:57.049764    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:58.792693 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:58.792656    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:58.986898 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:58:58.986864    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:00.078803 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.078768    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:00.148126 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.148095    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz"]
Oct 30 21:59:00.148371 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.148360    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296" containerName="util"
Oct 30 21:59:00.148414 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.148373    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296" containerName="util"
Oct 30 21:59:00.148414 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.148392    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296" containerName="extract"
Oct 30 21:59:00.148414 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.148398    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296" containerName="extract"
Oct 30 21:59:00.148502 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.148415    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296" containerName="pull"
Oct 30 21:59:00.148502 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.148420    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296" containerName="pull"
Oct 30 21:59:00.148502 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.148464    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c957ae-e3ab-4ec0-8dc6-e2ed8f88f296" containerName="extract"
Oct 30 21:59:00.150300 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.150283    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz"
Oct 30 21:59:00.203304 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.203277    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Oct 30 21:59:00.203304 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.203290    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Oct 30 21:59:00.210800 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.210776    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Oct 30 21:59:00.224352 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.224327    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-w6ncz\" (UID: \"342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz"
Oct 30 21:59:00.224475 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.224390    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6ksf\" (UniqueName: \"kubernetes.io/projected/342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9-kube-api-access-z6ksf\") pod \"keda-metrics-apiserver-86bd856cfb-w6ncz\" (UID: \"342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz"
Oct 30 21:59:00.224475 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.224452    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-w6ncz\" (UID: \"342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz"
Oct 30 21:59:00.248527 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.248505    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:00.253783 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.253761    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Oct 30 21:59:00.253895 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.253813    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Oct 30 21:59:00.253940 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.253896    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-ndw79\""
Oct 30 21:59:00.255044 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.255027    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz"]
Oct 30 21:59:00.325024 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.324997    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-w6ncz\" (UID: \"342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz"
Oct 30 21:59:00.325196 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.325043    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6ksf\" (UniqueName: \"kubernetes.io/projected/342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9-kube-api-access-z6ksf\") pod \"keda-metrics-apiserver-86bd856cfb-w6ncz\" (UID: \"342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz"
Oct 30 21:59:00.325196 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.325066    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-w6ncz\" (UID: \"342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz"
Oct 30 21:59:00.325384 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.325370    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-w6ncz\" (UID: \"342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz"
Oct 30 21:59:00.327617 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.327598    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-w6ncz\" (UID: \"342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz"
Oct 30 21:59:00.383818 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.383744    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6ksf\" (UniqueName: \"kubernetes.io/projected/342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9-kube-api-access-z6ksf\") pod \"keda-metrics-apiserver-86bd856cfb-w6ncz\" (UID: \"342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz"
Oct 30 21:59:00.460786 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.460749    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:00.461616 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.461598    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz"
Oct 30 21:59:00.615284 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.615251    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:00.680225 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.680194    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-shtgf"]
Oct 30 21:59:00.682218 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.682203    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-7b7c985c58-shtgf"
Oct 30 21:59:00.728157 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.728131    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4af254a3-9403-41c5-a811-c11ba8fdafea-certificates\") pod \"keda-admission-7b7c985c58-shtgf\" (UID: \"4af254a3-9403-41c5-a811-c11ba8fdafea\") " pod="openshift-keda/keda-admission-7b7c985c58-shtgf"
Oct 30 21:59:00.728157 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.728161    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cr6l\" (UniqueName: \"kubernetes.io/projected/4af254a3-9403-41c5-a811-c11ba8fdafea-kube-api-access-2cr6l\") pod \"keda-admission-7b7c985c58-shtgf\" (UID: \"4af254a3-9403-41c5-a811-c11ba8fdafea\") " pod="openshift-keda/keda-admission-7b7c985c58-shtgf"
Oct 30 21:59:00.737158 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.737139    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Oct 30 21:59:00.748546 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.748517    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-shtgf"]
Oct 30 21:59:00.796172 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.796140    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:00.829129 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.829100    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4af254a3-9403-41c5-a811-c11ba8fdafea-certificates\") pod \"keda-admission-7b7c985c58-shtgf\" (UID: \"4af254a3-9403-41c5-a811-c11ba8fdafea\") " pod="openshift-keda/keda-admission-7b7c985c58-shtgf"
Oct 30 21:59:00.829301 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.829133    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cr6l\" (UniqueName: \"kubernetes.io/projected/4af254a3-9403-41c5-a811-c11ba8fdafea-kube-api-access-2cr6l\") pod \"keda-admission-7b7c985c58-shtgf\" (UID: \"4af254a3-9403-41c5-a811-c11ba8fdafea\") " pod="openshift-keda/keda-admission-7b7c985c58-shtgf"
Oct 30 21:59:00.831648 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.831630    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4af254a3-9403-41c5-a811-c11ba8fdafea-certificates\") pod \"keda-admission-7b7c985c58-shtgf\" (UID: \"4af254a3-9403-41c5-a811-c11ba8fdafea\") " pod="openshift-keda/keda-admission-7b7c985c58-shtgf"
Oct 30 21:59:00.861983 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.861935    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz"]
Oct 30 21:59:00.864641 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:59:00.864614    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342000dd_5e3d_493c_bb4f_1c2e3b9ab2e9.slice/crio-3ad45f402ca598d1a2941772b41ad2692965579920cf80aea2db58b48112f4f9 WatchSource:0}: Error finding container 3ad45f402ca598d1a2941772b41ad2692965579920cf80aea2db58b48112f4f9: Status 404 returned error can't find the container with id 3ad45f402ca598d1a2941772b41ad2692965579920cf80aea2db58b48112f4f9
Oct 30 21:59:00.886511 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.886486    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cr6l\" (UniqueName: \"kubernetes.io/projected/4af254a3-9403-41c5-a811-c11ba8fdafea-kube-api-access-2cr6l\") pod \"keda-admission-7b7c985c58-shtgf\" (UID: \"4af254a3-9403-41c5-a811-c11ba8fdafea\") " pod="openshift-keda/keda-admission-7b7c985c58-shtgf"
Oct 30 21:59:00.963585 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.963511    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:00.993462 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:00.993434    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-7b7c985c58-shtgf"
Oct 30 21:59:01.275969 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:01.275862    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz" event={"ID":"342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9","Type":"ContainerStarted","Data":"3ad45f402ca598d1a2941772b41ad2692965579920cf80aea2db58b48112f4f9"}
Oct 30 21:59:01.361372 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:01.361259    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-shtgf"]
Oct 30 21:59:01.363846 ip-10-0-130-162 kubenswrapper[4473]: W1030 21:59:01.363819    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af254a3_9403_41c5_a811_c11ba8fdafea.slice/crio-0a73b9714b595782e80dd0db0767c54fbc169378e72b2b628d8d1ec949442a32 WatchSource:0}: Error finding container 0a73b9714b595782e80dd0db0767c54fbc169378e72b2b628d8d1ec949442a32: Status 404 returned error can't find the container with id 0a73b9714b595782e80dd0db0767c54fbc169378e72b2b628d8d1ec949442a32
Oct 30 21:59:02.280894 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:02.280865    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-7b7c985c58-shtgf" event={"ID":"4af254a3-9403-41c5-a811-c11ba8fdafea","Type":"ContainerStarted","Data":"0a73b9714b595782e80dd0db0767c54fbc169378e72b2b628d8d1ec949442a32"}
Oct 30 21:59:03.116068 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:03.116027    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:04.168074 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:04.168039    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:04.648038 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:04.648008    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:04.853083 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:04.853002    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:05.295113 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:05.295075    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-7b7c985c58-shtgf" event={"ID":"4af254a3-9403-41c5-a811-c11ba8fdafea","Type":"ContainerStarted","Data":"6368d467271f35292b383875bea3499ed08263ea998cc65d38d31cb81108221b"}
Oct 30 21:59:05.295623 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:05.295243    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-7b7c985c58-shtgf"
Oct 30 21:59:05.414290 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:05.414235    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-admission-7b7c985c58-shtgf" podStartSLOduration=2.36821246 podStartE2EDuration="5.41421917s" totalImagesPullingTime="3.04600671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:05.412799947 +0000 UTC m=+363.649009292" watchObservedRunningTime="2026-10-30 21:59:05.41421917 +0000 UTC m=+363.650428516"
Oct 30 21:59:06.301331 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:06.301295    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz" event={"ID":"342000dd-5e3d-493c-bb4f-1c2e3b9ab2e9","Type":"ContainerStarted","Data":"2c7b79d1f381aa9fdfb553514b4b1fe86d507a59e6f102caf326fe24fe0ec9b7"}
Oct 30 21:59:06.420381 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:06.420326    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz" podStartSLOduration=1.439953658 podStartE2EDuration="6.42030855s" totalImagesPullingTime="4.980354892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:06.420070164 +0000 UTC m=+364.656279510" watchObservedRunningTime="2026-10-30 21:59:06.42030855 +0000 UTC m=+364.656517886"
Oct 30 21:59:07.305876 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:07.305843    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz"
Oct 30 21:59:11.304537 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:59:11.304501    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 21:59:18.314928 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:18.314886    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-w6ncz"
Oct 30 21:59:19.017844 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:19.017812    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:26.304299 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:26.304271    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-7b7c985c58-shtgf"
Oct 30 21:59:41.303538 ip-10-0-130-162 kubenswrapper[4473]: E1030 21:59:41.303502    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 21:59:44.132690 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:44.132578    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:44.296471 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:44.296426    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:48.953715 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:48.953685    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:49.109994 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:49.109912    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:53.562484 ip-10-0-130-162 kubenswrapper[4473]: I1030 21:59:53.562447    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:11.303033 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:00:11.302984    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:00:19.185480 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.185440    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-85b8fbffd9-fwwrw"]
Oct 30 22:00:19.191155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.191118    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-85b8fbffd9-fwwrw"
Oct 30 22:00:19.224520 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.224490    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:19.248603 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.248575    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-85b8fbffd9-fwwrw"]
Oct 30 22:00:19.269546 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.269522    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5mj9\" (UniqueName: \"kubernetes.io/projected/668f3a5a-81ec-4603-abdf-793b02ce2b55-kube-api-access-w5mj9\") pod \"llmisvc-controller-manager-85b8fbffd9-fwwrw\" (UID: \"668f3a5a-81ec-4603-abdf-793b02ce2b55\") " pod="kserve/llmisvc-controller-manager-85b8fbffd9-fwwrw"
Oct 30 22:00:19.269670 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.269556    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/668f3a5a-81ec-4603-abdf-793b02ce2b55-cert\") pod \"llmisvc-controller-manager-85b8fbffd9-fwwrw\" (UID: \"668f3a5a-81ec-4603-abdf-793b02ce2b55\") " pod="kserve/llmisvc-controller-manager-85b8fbffd9-fwwrw"
Oct 30 22:00:19.297525 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.297499    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 30 22:00:19.297814 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.297796    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 30 22:00:19.297893 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.297796    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-trhjz\""
Oct 30 22:00:19.297893 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.297857    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Oct 30 22:00:19.369933 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.369900    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5mj9\" (UniqueName: \"kubernetes.io/projected/668f3a5a-81ec-4603-abdf-793b02ce2b55-kube-api-access-w5mj9\") pod \"llmisvc-controller-manager-85b8fbffd9-fwwrw\" (UID: \"668f3a5a-81ec-4603-abdf-793b02ce2b55\") " pod="kserve/llmisvc-controller-manager-85b8fbffd9-fwwrw"
Oct 30 22:00:19.369933 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.369938    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/668f3a5a-81ec-4603-abdf-793b02ce2b55-cert\") pod \"llmisvc-controller-manager-85b8fbffd9-fwwrw\" (UID: \"668f3a5a-81ec-4603-abdf-793b02ce2b55\") " pod="kserve/llmisvc-controller-manager-85b8fbffd9-fwwrw"
Oct 30 22:00:19.372689 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.372660    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/668f3a5a-81ec-4603-abdf-793b02ce2b55-cert\") pod \"llmisvc-controller-manager-85b8fbffd9-fwwrw\" (UID: \"668f3a5a-81ec-4603-abdf-793b02ce2b55\") " pod="kserve/llmisvc-controller-manager-85b8fbffd9-fwwrw"
Oct 30 22:00:19.428068 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.428040    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5mj9\" (UniqueName: \"kubernetes.io/projected/668f3a5a-81ec-4603-abdf-793b02ce2b55-kube-api-access-w5mj9\") pod \"llmisvc-controller-manager-85b8fbffd9-fwwrw\" (UID: \"668f3a5a-81ec-4603-abdf-793b02ce2b55\") " pod="kserve/llmisvc-controller-manager-85b8fbffd9-fwwrw"
Oct 30 22:00:19.487803 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.487723    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:19.505689 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.505667    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-85b8fbffd9-fwwrw"
Oct 30 22:00:19.729677 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.729648    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:19.875836 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:19.875815    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-85b8fbffd9-fwwrw"]
Oct 30 22:00:19.877939 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:00:19.877910    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668f3a5a_81ec_4603_abdf_793b02ce2b55.slice/crio-5b92c58a79db48235b654a98fe182aa9c2fecc567a1faf8df3f4602fda18413d WatchSource:0}: Error finding container 5b92c58a79db48235b654a98fe182aa9c2fecc567a1faf8df3f4602fda18413d: Status 404 returned error can't find the container with id 5b92c58a79db48235b654a98fe182aa9c2fecc567a1faf8df3f4602fda18413d
Oct 30 22:00:20.193316 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:20.193285    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:20.385644 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:20.385613    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:20.580421 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:20.580338    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-85b8fbffd9-fwwrw" event={"ID":"668f3a5a-81ec-4603-abdf-793b02ce2b55","Type":"ContainerStarted","Data":"5b92c58a79db48235b654a98fe182aa9c2fecc567a1faf8df3f4602fda18413d"}
Oct 30 22:00:20.795847 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:20.795817    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:20.992571 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:20.992536    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:21.178071 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:21.178039    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:26.606443 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:26.606414    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-85b8fbffd9-fwwrw" event={"ID":"668f3a5a-81ec-4603-abdf-793b02ce2b55","Type":"ContainerStarted","Data":"7eea1d981e819b5612fea9f13aaf3080c71b30074fb33c2f04057ec1884b813d"}
Oct 30 22:00:26.606850 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:26.606485    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-85b8fbffd9-fwwrw"
Oct 30 22:00:26.722644 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:26.722585    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-85b8fbffd9-fwwrw" podStartSLOduration=1.064678521 podStartE2EDuration="7.722572089s" totalImagesPullingTime="6.657893568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:26.72215663 +0000 UTC m=+444.958365976" watchObservedRunningTime="2026-10-30 22:00:26.722572089 +0000 UTC m=+444.958781436"
Oct 30 22:00:41.303599 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:00:41.303555    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:00:53.916752 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:53.916721    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:57.612744 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:00:57.612713    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-85b8fbffd9-fwwrw"
Oct 30 22:01:10.968693 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:10.968656    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:11.303917 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:01:11.303835    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:01:31.740125 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:31.740042    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:41.303163 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:01:41.303127    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:01:47.687707 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:47.687669    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:47.787969 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:47.787913    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-56ddc4c448-l2pvj"]
Oct 30 22:01:47.791480 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:47.791459    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-l2pvj"
Oct 30 22:01:47.861677 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:47.861648    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:47.871444 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:47.871424    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsksw\" (UniqueName: \"kubernetes.io/projected/488a4af2-6299-4e6d-b5fc-4ff22ef068f4-kube-api-access-hsksw\") pod \"model-serving-api-56ddc4c448-l2pvj\" (UID: \"488a4af2-6299-4e6d-b5fc-4ff22ef068f4\") " pod="kserve/model-serving-api-56ddc4c448-l2pvj"
Oct 30 22:01:47.871555 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:47.871478    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488a4af2-6299-4e6d-b5fc-4ff22ef068f4-tls-certs\") pod \"model-serving-api-56ddc4c448-l2pvj\" (UID: \"488a4af2-6299-4e6d-b5fc-4ff22ef068f4\") " pod="kserve/model-serving-api-56ddc4c448-l2pvj"
Oct 30 22:01:47.897034 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:47.897007    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Oct 30 22:01:47.898368 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:47.898350    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-l2pvj"]
Oct 30 22:01:47.947553 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:47.947489    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-6xvs4\""
Oct 30 22:01:47.972123 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:47.972098    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488a4af2-6299-4e6d-b5fc-4ff22ef068f4-tls-certs\") pod \"model-serving-api-56ddc4c448-l2pvj\" (UID: \"488a4af2-6299-4e6d-b5fc-4ff22ef068f4\") " pod="kserve/model-serving-api-56ddc4c448-l2pvj"
Oct 30 22:01:47.972254 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:47.972144    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsksw\" (UniqueName: \"kubernetes.io/projected/488a4af2-6299-4e6d-b5fc-4ff22ef068f4-kube-api-access-hsksw\") pod \"model-serving-api-56ddc4c448-l2pvj\" (UID: \"488a4af2-6299-4e6d-b5fc-4ff22ef068f4\") " pod="kserve/model-serving-api-56ddc4c448-l2pvj"
Oct 30 22:01:47.974607 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:47.974587    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488a4af2-6299-4e6d-b5fc-4ff22ef068f4-tls-certs\") pod \"model-serving-api-56ddc4c448-l2pvj\" (UID: \"488a4af2-6299-4e6d-b5fc-4ff22ef068f4\") " pod="kserve/model-serving-api-56ddc4c448-l2pvj"
Oct 30 22:01:48.032132 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:48.032083    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:48.052713 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:48.052684    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsksw\" (UniqueName: \"kubernetes.io/projected/488a4af2-6299-4e6d-b5fc-4ff22ef068f4-kube-api-access-hsksw\") pod \"model-serving-api-56ddc4c448-l2pvj\" (UID: \"488a4af2-6299-4e6d-b5fc-4ff22ef068f4\") " pod="kserve/model-serving-api-56ddc4c448-l2pvj"
Oct 30 22:01:48.106719 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:48.106693    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-l2pvj"
Oct 30 22:01:48.230880 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:48.230772    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:48.476805 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:48.476716    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-l2pvj"]
Oct 30 22:01:48.479423 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:01:48.479400    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488a4af2_6299_4e6d_b5fc_4ff22ef068f4.slice/crio-89c5a050c3a0a38cfc271cc601bacc87eef5ce552bc1fd2b09271849893d4320 WatchSource:0}: Error finding container 89c5a050c3a0a38cfc271cc601bacc87eef5ce552bc1fd2b09271849893d4320: Status 404 returned error can't find the container with id 89c5a050c3a0a38cfc271cc601bacc87eef5ce552bc1fd2b09271849893d4320
Oct 30 22:01:48.899693 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:48.899663    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-l2pvj" event={"ID":"488a4af2-6299-4e6d-b5fc-4ff22ef068f4","Type":"ContainerStarted","Data":"89c5a050c3a0a38cfc271cc601bacc87eef5ce552bc1fd2b09271849893d4320"}
Oct 30 22:01:50.909491 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:50.909456    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-l2pvj" event={"ID":"488a4af2-6299-4e6d-b5fc-4ff22ef068f4","Type":"ContainerStarted","Data":"94b1d37b41c86823cccc40d5c8fc011283d1e5530accc88849f26ccefa3e3794"}
Oct 30 22:01:50.909874 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:50.909580    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-56ddc4c448-l2pvj"
Oct 30 22:01:51.099166 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:51.099065    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/model-serving-api-56ddc4c448-l2pvj" podStartSLOduration=1.756873151 podStartE2EDuration="4.09905145s" totalImagesPullingTime="2.342178299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:51.047660767 +0000 UTC m=+529.283870234" watchObservedRunningTime="2026-10-30 22:01:51.09905145 +0000 UTC m=+529.335260808"
Oct 30 22:01:52.716302 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:52.716268    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:52.871007 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:52.870945    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:54.265829 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:54.265738    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:59.656039 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:59.655994    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:59.832641 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:01:59.832592    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:01.918056 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:02:01.918023    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-56ddc4c448-l2pvj"
Oct 30 22:02:11.302991 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:02:11.302934    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:02:41.303440 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:02:41.303340    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:02:54.590378 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:02:54.590300    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:01.906171 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:03:01.906143    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:03:01.922699 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:03:01.922672    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:03:11.303921 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:03:11.303879    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:03:29.066887 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:03:29.066838    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:41.303618 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:03:41.303578    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:03:42.149418 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:03:42.149385    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:42.309422 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:03:42.309386    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:54.956867 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:03:54.956835    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:11.303611 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:04:11.303572    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:04:18.044612 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:18.044534    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:18.243230 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:18.243191    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:18.469225 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:18.469193    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:35.748476 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:35.748440    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:35.926133 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:35.926102    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:41.305793 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:04:41.305749    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:04:47.530930 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:47.530896    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:51.949101 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:51.949065    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:52.211198 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:52.211117    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:55.312009 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:55.311976    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:55.593302 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:55.593216    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:55.773744 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:55.773708    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:56.306683 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.306650    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"]
Oct 30 22:04:56.309039 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.309023    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:04:56.333938 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.333914    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-68fb69745b-lnzhw\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:04:56.334276 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.333946    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-68fb69745b-lnzhw\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:04:56.334276 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.334037    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-68fb69745b-lnzhw\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:04:56.334276 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.334088    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45gjf\" (UniqueName: \"kubernetes.io/projected/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kube-api-access-45gjf\") pod \"isvc-sklearn-batcher-predictor-68fb69745b-lnzhw\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:04:56.334276 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.334110    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-68fb69745b-lnzhw\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:04:56.362942 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.362918    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"openshift-service-ca.crt\""
Oct 30 22:04:56.413510 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.413487    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"seaweedfs-s3-creds\""
Oct 30 22:04:56.413729 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.413711    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Oct 30 22:04:56.413803 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.413712    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Oct 30 22:04:56.413803 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.413743    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:04:56.413803 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.413767    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"default-dockercfg-4v4s8\""
Oct 30 22:04:56.413803 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.413797    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"kube-root-ca.crt\""
Oct 30 22:04:56.435232 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.435204    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-68fb69745b-lnzhw\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:04:56.435352 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.435236    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-68fb69745b-lnzhw\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:04:56.435352 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.435266    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-68fb69745b-lnzhw\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:04:56.435352 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.435330    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45gjf\" (UniqueName: \"kubernetes.io/projected/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kube-api-access-45gjf\") pod \"isvc-sklearn-batcher-predictor-68fb69745b-lnzhw\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:04:56.435494 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.435361    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-68fb69745b-lnzhw\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:04:56.435798 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.435775    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-68fb69745b-lnzhw\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:04:56.435968 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.435930    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-68fb69745b-lnzhw\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:04:56.436020 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.435995    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-68fb69745b-lnzhw\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:04:56.437839 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.437824    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-68fb69745b-lnzhw\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:04:56.465672 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.465641    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"]
Oct 30 22:04:56.536110 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.536085    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45gjf\" (UniqueName: \"kubernetes.io/projected/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kube-api-access-45gjf\") pod \"isvc-sklearn-batcher-predictor-68fb69745b-lnzhw\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:04:56.623241 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.623161    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:04:56.699999 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.699945    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:56.952735 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:56.952703    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:57.052596 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:57.052564    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"]
Oct 30 22:04:57.054707 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:04:57.054681    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c32f0d5_2bdf_4732_a6c9_6dd19aab71f8.slice/crio-6efc7feb37e20c22927544ca1583c742a830d2e32dd13a1924b1c51058148c65 WatchSource:0}: Error finding container 6efc7feb37e20c22927544ca1583c742a830d2e32dd13a1924b1c51058148c65: Status 404 returned error can't find the container with id 6efc7feb37e20c22927544ca1583c742a830d2e32dd13a1924b1c51058148c65
Oct 30 22:04:57.056695 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:57.056676    4473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:04:57.596856 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:04:57.596820    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" event={"ID":"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8","Type":"ContainerStarted","Data":"6efc7feb37e20c22927544ca1583c742a830d2e32dd13a1924b1c51058148c65"}
Oct 30 22:05:01.614678 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:01.614647    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" event={"ID":"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8","Type":"ContainerStarted","Data":"8bb988efdd10cd7ce3b57563f2bbb9751d2d6efbadfbdd9988f02c553bb26d31"}
Oct 30 22:05:06.635698 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:06.635664    4473 generic.go:356] "Generic (PLEG): container finished" podID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerID="8bb988efdd10cd7ce3b57563f2bbb9751d2d6efbadfbdd9988f02c553bb26d31" exitCode=0
Oct 30 22:05:06.636121 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:06.635745    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" event={"ID":"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8","Type":"ContainerDied","Data":"8bb988efdd10cd7ce3b57563f2bbb9751d2d6efbadfbdd9988f02c553bb26d31"}
Oct 30 22:05:11.311511 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:05:11.311472    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:05:22.718491 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:22.718453    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" event={"ID":"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8","Type":"ContainerStarted","Data":"7d66e388fdb3d9d180d61d58c055a1da380d0d38818d93cc79c757bf969ef11a"}
Oct 30 22:05:24.729061 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:24.728976    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" event={"ID":"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8","Type":"ContainerStarted","Data":"d7078225988c33202963b77ae67301a6c9d92c1a19c9e19af508592685a7e2ce"}
Oct 30 22:05:34.773216 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:34.773125    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" event={"ID":"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8","Type":"ContainerStarted","Data":"d8761da229e767b0603632a6729c539fe0330146230c4aebc34e696b183f5bbb"}
Oct 30 22:05:34.773674 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:34.773406    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:05:34.773674 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:34.773457    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:05:34.774858 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:34.774825    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Oct 30 22:05:34.778452 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:34.778433    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:05:35.000231 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:35.000179    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podStartSLOduration=2.809355535 podStartE2EDuration="39.000164639s" totalImagesPullingTime="32.034677405s" totalInitContainerRuntime="4.156131699s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:56 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:34.947707082 +0000 UTC m=+753.183916429" watchObservedRunningTime="2026-10-30 22:05:35.000164639 +0000 UTC m=+753.236373983"
Oct 30 22:05:35.777188 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:35.777154    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:05:35.777573 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:35.777278    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Oct 30 22:05:35.777940 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:35.777919    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:05:36.780833 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:36.780794    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Oct 30 22:05:36.781280 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:36.781082    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:05:41.303300 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:05:41.303266    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:05:46.781318 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:46.781232    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Oct 30 22:05:46.781765 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:46.781644    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:05:55.604967 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:55.604869    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:56.780946 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:56.780903    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Oct 30 22:05:56.781462 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:05:56.781438    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:06:06.780831 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:06:06.780782    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Oct 30 22:06:06.781300 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:06:06.781273    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:06:11.304268 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:06:11.304232    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:06:16.780821 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:06:16.780781    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Oct 30 22:06:16.781314 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:06:16.781287    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:06:26.781617 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:06:26.781571    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Oct 30 22:06:26.782183 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:06:26.782110    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:06:36.780841 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:06:36.780788    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Oct 30 22:06:36.781303 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:06:36.781248    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:06:41.303532 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:06:41.303492    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:06:46.782195 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:06:46.782160    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:06:46.782673 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:06:46.782265    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:06:55.946171 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:06:55.946137    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:58.144877 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:06:58.144840    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:01.310887 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:01.310851    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"]
Oct 30 22:07:01.311363 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:01.311140    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="agent" containerID="cri-o://d8761da229e767b0603632a6729c539fe0330146230c4aebc34e696b183f5bbb" gracePeriod=30
Oct 30 22:07:01.311363 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:01.311140    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kserve-container" containerID="cri-o://7d66e388fdb3d9d180d61d58c055a1da380d0d38818d93cc79c757bf969ef11a" gracePeriod=30
Oct 30 22:07:01.311363 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:01.311225    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kube-rbac-proxy" containerID="cri-o://d7078225988c33202963b77ae67301a6c9d92c1a19c9e19af508592685a7e2ce" gracePeriod=30
Oct 30 22:07:01.318090 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:01.318061    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:02.127823 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:02.127791    4473 generic.go:356] "Generic (PLEG): container finished" podID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerID="d7078225988c33202963b77ae67301a6c9d92c1a19c9e19af508592685a7e2ce" exitCode=0
Oct 30 22:07:02.128022 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:02.127865    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" event={"ID":"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8","Type":"ContainerDied","Data":"d7078225988c33202963b77ae67301a6c9d92c1a19c9e19af508592685a7e2ce"}
Oct 30 22:07:04.449967 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:04.449919    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:04.678116 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:04.678080    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:04.774252 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:04.774168    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Oct 30 22:07:05.163517 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.163488    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"]
Oct 30 22:07:05.167130 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.167112    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:05.196813 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.196784    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lcw8\" (UniqueName: \"kubernetes.io/projected/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kube-api-access-9lcw8\") pod \"isvc-primary-0052a3-predictor-78b6456644-g8m27\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") " pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:05.196992 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.196967    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-cabundle-cert\") pod \"isvc-primary-0052a3-predictor-78b6456644-g8m27\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") " pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:05.197077 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.197060    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-0052a3-predictor-78b6456644-g8m27\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") " pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:05.197148 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.197092    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kserve-provision-location\") pod \"isvc-primary-0052a3-predictor-78b6456644-g8m27\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") " pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:05.197148 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.197121    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-proxy-tls\") pod \"isvc-primary-0052a3-predictor-78b6456644-g8m27\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") " pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:05.221114 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.221096    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-primary-0052a3-kube-rbac-proxy-sar-config\""
Oct 30 22:07:05.271843 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.271815    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:07:05.272161 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.272101    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-primary-0052a3-predictor-serving-cert\""
Oct 30 22:07:05.272161 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.272122    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:07:05.272161 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.272125    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:07:05.272383 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.272162    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:07:05.272383 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.272274    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:07:05.273932 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.273909    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"]
Oct 30 22:07:05.297709 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.297682    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-cabundle-cert\") pod \"isvc-primary-0052a3-predictor-78b6456644-g8m27\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") " pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:05.297813 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.297718    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-0052a3-predictor-78b6456644-g8m27\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") " pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:05.297813 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.297739    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kserve-provision-location\") pod \"isvc-primary-0052a3-predictor-78b6456644-g8m27\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") " pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:05.297813 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.297764    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-proxy-tls\") pod \"isvc-primary-0052a3-predictor-78b6456644-g8m27\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") " pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:05.297984 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.297895    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lcw8\" (UniqueName: \"kubernetes.io/projected/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kube-api-access-9lcw8\") pod \"isvc-primary-0052a3-predictor-78b6456644-g8m27\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") " pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:05.298188 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.298167    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kserve-provision-location\") pod \"isvc-primary-0052a3-predictor-78b6456644-g8m27\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") " pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:05.298491 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.298467    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-0052a3-predictor-78b6456644-g8m27\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") " pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:05.298605 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.298495    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-cabundle-cert\") pod \"isvc-primary-0052a3-predictor-78b6456644-g8m27\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") " pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:05.300470 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.300449    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-proxy-tls\") pod \"isvc-primary-0052a3-predictor-78b6456644-g8m27\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") " pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:05.356072 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.356042    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lcw8\" (UniqueName: \"kubernetes.io/projected/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kube-api-access-9lcw8\") pod \"isvc-primary-0052a3-predictor-78b6456644-g8m27\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") " pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:05.505761 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.505671    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:05.930312 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:05.930293    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"]
Oct 30 22:07:05.932738 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:07:05.932710    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85912ff_a3f2_4c2c_9d92_2883e01d5b8d.slice/crio-9d33c199cdcecc7b091326052a0d43c5411effd8e320e60e0c4e556fe698f8a4 WatchSource:0}: Error finding container 9d33c199cdcecc7b091326052a0d43c5411effd8e320e60e0c4e556fe698f8a4: Status 404 returned error can't find the container with id 9d33c199cdcecc7b091326052a0d43c5411effd8e320e60e0c4e556fe698f8a4
Oct 30 22:07:06.144706 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:06.144669    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" event={"ID":"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d","Type":"ContainerStarted","Data":"d28b0d8081196df1b9d4a4d8517cc3161bc748c76c23cf9271737a73a813dee1"}
Oct 30 22:07:06.144877 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:06.144722    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" event={"ID":"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d","Type":"ContainerStarted","Data":"9d33c199cdcecc7b091326052a0d43c5411effd8e320e60e0c4e556fe698f8a4"}
Oct 30 22:07:06.781416 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:06.781363    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Oct 30 22:07:06.781884 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:06.781647    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:07:07.151594 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:07.151565    4473 generic.go:356] "Generic (PLEG): container finished" podID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerID="7d66e388fdb3d9d180d61d58c055a1da380d0d38818d93cc79c757bf969ef11a" exitCode=0
Oct 30 22:07:07.151776 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:07.151650    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" event={"ID":"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8","Type":"ContainerDied","Data":"7d66e388fdb3d9d180d61d58c055a1da380d0d38818d93cc79c757bf969ef11a"}
Oct 30 22:07:09.774036 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:09.773995    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Oct 30 22:07:10.165250 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:10.165216    4473 generic.go:356] "Generic (PLEG): container finished" podID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerID="d28b0d8081196df1b9d4a4d8517cc3161bc748c76c23cf9271737a73a813dee1" exitCode=0
Oct 30 22:07:10.165418 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:10.165258    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" event={"ID":"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d","Type":"ContainerDied","Data":"d28b0d8081196df1b9d4a4d8517cc3161bc748c76c23cf9271737a73a813dee1"}
Oct 30 22:07:11.171861 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:11.171827    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" event={"ID":"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d","Type":"ContainerStarted","Data":"e0cb459be318d5d3b123d146fe291288266b10b5621d17bac1bff5e23fd7841d"}
Oct 30 22:07:11.171861 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:11.171885    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" event={"ID":"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d","Type":"ContainerStarted","Data":"729980ed09a58e75b916064e4d43db69b5cb0bd33a43de07be08d33dd831864c"}
Oct 30 22:07:11.172481 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:11.172141    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:11.304113 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:07:11.304077    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:07:11.343888 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:11.343841    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" podStartSLOduration=2.26922185 podStartE2EDuration="6.343827783s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.074605933s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:11.342532963 +0000 UTC m=+849.578742309" watchObservedRunningTime="2026-10-30 22:07:11.343827783 +0000 UTC m=+849.580037128"
Oct 30 22:07:12.176215 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:12.176185    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:12.177264 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:12.177236    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 22:07:13.180240 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:13.180200    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 22:07:14.774433 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:14.774389    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Oct 30 22:07:14.774923 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:14.774544    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:07:16.781474 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:16.781431    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Oct 30 22:07:16.781984 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:16.781771    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:07:18.184963 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:18.184919    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:07:18.185528 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:18.185501    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 22:07:19.774268 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:19.774232    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Oct 30 22:07:24.773968 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:24.773907    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Oct 30 22:07:26.780748 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:26.780688    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Oct 30 22:07:26.781202 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:26.780866    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:07:26.781202 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:26.781021    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:07:26.781202 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:26.781145    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:07:28.186426 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:28.186390    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 22:07:29.773618 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:29.773582    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Oct 30 22:07:31.974215 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:31.974188    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:07:32.017326 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.017298    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kube-api-access-45gjf\" (UniqueName: \"kubernetes.io/projected/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kube-api-access-45gjf\") pod \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") "
Oct 30 22:07:32.017414 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.017362    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kserve-provision-location\") pod \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") "
Oct 30 22:07:32.017414 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.017388    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kube-rbac-proxy-sar-config\") pod \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") "
Oct 30 22:07:32.017499 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.017420    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-proxy-tls\") pod \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") "
Oct 30 22:07:32.017499 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.017438    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-cabundle-cert\") pod \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\" (UID: \"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8\") "
Oct 30 22:07:32.017794 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.017713    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kserve-provision-location" pod "8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" (UID: "8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:32.017988 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.017721    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kube-rbac-proxy-sar-config" pod "8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" (UID: "8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:32.018051 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.017908    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-cabundle-cert" pod "8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" (UID: "8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:32.019748 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.019725    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kube-api-access-45gjf" pod "8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" (UID: "8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8"). InnerVolumeSpecName "kube-api-access-45gjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:32.019748 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.019734    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-proxy-tls" pod "8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" (UID: "8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:32.118155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.118078    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45gjf\" (UniqueName: \"kubernetes.io/projected/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kube-api-access-45gjf\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:32.118155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.118103    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:32.118155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.118112    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:32.118155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.118121    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:32.118155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.118129    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:32.261254 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.261214    4473 generic.go:356] "Generic (PLEG): container finished" podID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerID="d8761da229e767b0603632a6729c539fe0330146230c4aebc34e696b183f5bbb" exitCode=0
Oct 30 22:07:32.261413 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.261283    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"
Oct 30 22:07:32.261413 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.261298    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" event={"ID":"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8","Type":"ContainerDied","Data":"d8761da229e767b0603632a6729c539fe0330146230c4aebc34e696b183f5bbb"}
Oct 30 22:07:32.261413 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.261374    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw" event={"ID":"8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8","Type":"ContainerDied","Data":"6efc7feb37e20c22927544ca1583c742a830d2e32dd13a1924b1c51058148c65"}
Oct 30 22:07:32.261413 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.261409    4473 scope.go:122] "RemoveContainer" containerID="d8761da229e767b0603632a6729c539fe0330146230c4aebc34e696b183f5bbb"
Oct 30 22:07:32.269822 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.269799    4473 scope.go:122] "RemoveContainer" containerID="d7078225988c33202963b77ae67301a6c9d92c1a19c9e19af508592685a7e2ce"
Oct 30 22:07:32.277384 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.277363    4473 scope.go:122] "RemoveContainer" containerID="7d66e388fdb3d9d180d61d58c055a1da380d0d38818d93cc79c757bf969ef11a"
Oct 30 22:07:32.284422 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.284404    4473 scope.go:122] "RemoveContainer" containerID="8bb988efdd10cd7ce3b57563f2bbb9751d2d6efbadfbdd9988f02c553bb26d31"
Oct 30 22:07:32.291700 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.291681    4473 scope.go:122] "RemoveContainer" containerID="d8761da229e767b0603632a6729c539fe0330146230c4aebc34e696b183f5bbb"
Oct 30 22:07:32.292007 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:07:32.291941    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8761da229e767b0603632a6729c539fe0330146230c4aebc34e696b183f5bbb\": container with ID starting with d8761da229e767b0603632a6729c539fe0330146230c4aebc34e696b183f5bbb not found: ID does not exist" containerID="d8761da229e767b0603632a6729c539fe0330146230c4aebc34e696b183f5bbb"
Oct 30 22:07:32.292094 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.292021    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8761da229e767b0603632a6729c539fe0330146230c4aebc34e696b183f5bbb"} err="failed to get container status \"d8761da229e767b0603632a6729c539fe0330146230c4aebc34e696b183f5bbb\": rpc error: code = NotFound desc = could not find container \"d8761da229e767b0603632a6729c539fe0330146230c4aebc34e696b183f5bbb\": container with ID starting with d8761da229e767b0603632a6729c539fe0330146230c4aebc34e696b183f5bbb not found: ID does not exist"
Oct 30 22:07:32.292094 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.292046    4473 scope.go:122] "RemoveContainer" containerID="d7078225988c33202963b77ae67301a6c9d92c1a19c9e19af508592685a7e2ce"
Oct 30 22:07:32.292311 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:07:32.292291    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7078225988c33202963b77ae67301a6c9d92c1a19c9e19af508592685a7e2ce\": container with ID starting with d7078225988c33202963b77ae67301a6c9d92c1a19c9e19af508592685a7e2ce not found: ID does not exist" containerID="d7078225988c33202963b77ae67301a6c9d92c1a19c9e19af508592685a7e2ce"
Oct 30 22:07:32.292374 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.292321    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7078225988c33202963b77ae67301a6c9d92c1a19c9e19af508592685a7e2ce"} err="failed to get container status \"d7078225988c33202963b77ae67301a6c9d92c1a19c9e19af508592685a7e2ce\": rpc error: code = NotFound desc = could not find container \"d7078225988c33202963b77ae67301a6c9d92c1a19c9e19af508592685a7e2ce\": container with ID starting with d7078225988c33202963b77ae67301a6c9d92c1a19c9e19af508592685a7e2ce not found: ID does not exist"
Oct 30 22:07:32.292374 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.292344    4473 scope.go:122] "RemoveContainer" containerID="7d66e388fdb3d9d180d61d58c055a1da380d0d38818d93cc79c757bf969ef11a"
Oct 30 22:07:32.292631 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:07:32.292605    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d66e388fdb3d9d180d61d58c055a1da380d0d38818d93cc79c757bf969ef11a\": container with ID starting with 7d66e388fdb3d9d180d61d58c055a1da380d0d38818d93cc79c757bf969ef11a not found: ID does not exist" containerID="7d66e388fdb3d9d180d61d58c055a1da380d0d38818d93cc79c757bf969ef11a"
Oct 30 22:07:32.292681 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.292627    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d66e388fdb3d9d180d61d58c055a1da380d0d38818d93cc79c757bf969ef11a"} err="failed to get container status \"7d66e388fdb3d9d180d61d58c055a1da380d0d38818d93cc79c757bf969ef11a\": rpc error: code = NotFound desc = could not find container \"7d66e388fdb3d9d180d61d58c055a1da380d0d38818d93cc79c757bf969ef11a\": container with ID starting with 7d66e388fdb3d9d180d61d58c055a1da380d0d38818d93cc79c757bf969ef11a not found: ID does not exist"
Oct 30 22:07:32.292681 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.292639    4473 scope.go:122] "RemoveContainer" containerID="8bb988efdd10cd7ce3b57563f2bbb9751d2d6efbadfbdd9988f02c553bb26d31"
Oct 30 22:07:32.292887 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:07:32.292869    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb988efdd10cd7ce3b57563f2bbb9751d2d6efbadfbdd9988f02c553bb26d31\": container with ID starting with 8bb988efdd10cd7ce3b57563f2bbb9751d2d6efbadfbdd9988f02c553bb26d31 not found: ID does not exist" containerID="8bb988efdd10cd7ce3b57563f2bbb9751d2d6efbadfbdd9988f02c553bb26d31"
Oct 30 22:07:32.292980 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.292895    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb988efdd10cd7ce3b57563f2bbb9751d2d6efbadfbdd9988f02c553bb26d31"} err="failed to get container status \"8bb988efdd10cd7ce3b57563f2bbb9751d2d6efbadfbdd9988f02c553bb26d31\": rpc error: code = NotFound desc = could not find container \"8bb988efdd10cd7ce3b57563f2bbb9751d2d6efbadfbdd9988f02c553bb26d31\": container with ID starting with 8bb988efdd10cd7ce3b57563f2bbb9751d2d6efbadfbdd9988f02c553bb26d31 not found: ID does not exist"
Oct 30 22:07:32.430042 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.430008    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"]
Oct 30 22:07:32.479475 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:32.479441    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-sklearn-batcher-predictor-68fb69745b-lnzhw"]
Oct 30 22:07:33.886389 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:33.886359    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" path="/var/lib/kubelet/pods/8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8/volumes"
Oct 30 22:07:34.167050 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.167019    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:34.474991 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.474905    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:34.833579 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.833502    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"]
Oct 30 22:07:34.833827 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.833816    4473 cpu_manager.go:414] "containerMap: removing container" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="agent"
Oct 30 22:07:34.833868 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.833829    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="agent"
Oct 30 22:07:34.833868 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.833839    4473 cpu_manager.go:414] "containerMap: removing container" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="storage-initializer"
Oct 30 22:07:34.833868 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.833844    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="storage-initializer"
Oct 30 22:07:34.833868 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.833858    4473 cpu_manager.go:414] "containerMap: removing container" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kube-rbac-proxy"
Oct 30 22:07:34.833868 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.833863    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kube-rbac-proxy"
Oct 30 22:07:34.834046 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.833871    4473 cpu_manager.go:414] "containerMap: removing container" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kserve-container"
Oct 30 22:07:34.834046 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.833875    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kserve-container"
Oct 30 22:07:34.834046 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.833929    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kube-rbac-proxy"
Oct 30 22:07:34.834046 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.833937    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="kserve-container"
Oct 30 22:07:34.834046 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.833962    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c32f0d5-2bdf-4732-a6c9-6dd19aab71f8" containerName="agent"
Oct 30 22:07:34.839131 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.839114    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:34.893387 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.893361    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:07:34.893387 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.893361    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"openshift-service-ca.crt\""
Oct 30 22:07:34.941833 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.941804    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:34.942008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.941858    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bdck\" (UniqueName: \"kubernetes.io/projected/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kube-api-access-5bdck\") pod \"isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:34.942008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.941961    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:34.942095 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.942010    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:34.942095 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.942039    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:34.943017 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.942991    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Oct 30 22:07:34.943145 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.943071    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"seaweedfs-s3-creds\""
Oct 30 22:07:34.943145 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.943089    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"default-dockercfg-4v4s8\""
Oct 30 22:07:34.943145 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.943115    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"kube-root-ca.crt\""
Oct 30 22:07:34.943304 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.943152    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Oct 30 22:07:34.945035 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:34.945018    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"]
Oct 30 22:07:35.042691 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:35.042658    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:35.042872 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:35.042703    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bdck\" (UniqueName: \"kubernetes.io/projected/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kube-api-access-5bdck\") pod \"isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:35.042872 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:35.042822    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:35.042872 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:35.042860    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:35.043069 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:35.042890    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:35.043158 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:35.043141    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:35.043618 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:35.043594    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:35.043618 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:35.043607    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:35.045367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:35.045342    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:35.101277 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:35.101204    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bdck\" (UniqueName: \"kubernetes.io/projected/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kube-api-access-5bdck\") pod \"isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") " pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:35.157702 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:35.157670    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:35.585144 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:35.585103    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"]
Oct 30 22:07:35.587290 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:07:35.587265    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa3e75b0_ddd8_481d_9bca_2e10f0b28061.slice/crio-f49f3fc609581a265e46cbf0120ad3509ef10cdb319cc59b98d5fc0af49a56be WatchSource:0}: Error finding container f49f3fc609581a265e46cbf0120ad3509ef10cdb319cc59b98d5fc0af49a56be: Status 404 returned error can't find the container with id f49f3fc609581a265e46cbf0120ad3509ef10cdb319cc59b98d5fc0af49a56be
Oct 30 22:07:36.283776 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:36.283741    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" event={"ID":"fa3e75b0-ddd8-481d-9bca-2e10f0b28061","Type":"ContainerStarted","Data":"fba8661085b93b093ea7cc1184b864a041d8d9c5366ee751e3c5acf0ab74e2ea"}
Oct 30 22:07:36.284196 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:36.283804    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" event={"ID":"fa3e75b0-ddd8-481d-9bca-2e10f0b28061","Type":"ContainerStarted","Data":"f49f3fc609581a265e46cbf0120ad3509ef10cdb319cc59b98d5fc0af49a56be"}
Oct 30 22:07:38.185789 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:38.185747    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 22:07:40.300923 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:40.300887    4473 generic.go:356] "Generic (PLEG): container finished" podID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerID="fba8661085b93b093ea7cc1184b864a041d8d9c5366ee751e3c5acf0ab74e2ea" exitCode=0
Oct 30 22:07:40.301391 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:40.300944    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" event={"ID":"fa3e75b0-ddd8-481d-9bca-2e10f0b28061","Type":"ContainerDied","Data":"fba8661085b93b093ea7cc1184b864a041d8d9c5366ee751e3c5acf0ab74e2ea"}
Oct 30 22:07:41.304274 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:07:41.304162    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:07:41.326882 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:41.307434    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" event={"ID":"fa3e75b0-ddd8-481d-9bca-2e10f0b28061","Type":"ContainerStarted","Data":"ff901a7099a1ec48733fb834cdb168dcac5d9a738c2d88547b3e72e1f17eca85"}
Oct 30 22:07:41.326882 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:41.307491    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" event={"ID":"fa3e75b0-ddd8-481d-9bca-2e10f0b28061","Type":"ContainerStarted","Data":"959e027195ba0d43ecda0574d81a2fa74bd97d873ef50f84beebccf0bbf2c480"}
Oct 30 22:07:41.326882 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:41.307516    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" event={"ID":"fa3e75b0-ddd8-481d-9bca-2e10f0b28061","Type":"ContainerStarted","Data":"e329ed3988a516737d9d3749c5426c6207435998444628c38276e5fe8338289b"}
Oct 30 22:07:41.326882 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:41.307768    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:41.531722 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:41.531667    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podStartSLOduration=3.592064371 podStartE2EDuration="7.53165034s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.939585969s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:41.5304404 +0000 UTC m=+879.766649748" watchObservedRunningTime="2026-10-30 22:07:41.53165034 +0000 UTC m=+879.767859681"
Oct 30 22:07:42.311848 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:42.311813    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:42.312298 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:42.311885    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:42.313077 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:42.313048    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Oct 30 22:07:42.313803 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:42.313781    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:07:43.315892 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:43.315855    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Oct 30 22:07:43.316367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:43.316344    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:07:48.185484 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:48.185444    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 22:07:48.320133 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:48.320108    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:07:48.320648 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:48.320614    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Oct 30 22:07:48.321080 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:48.321059    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:07:56.291585 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:56.291550    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:58.186045 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:58.185996    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 22:07:58.320914 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:58.320875    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Oct 30 22:07:58.321334 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:07:58.321309    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:08:01.955037 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:01.955015    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:08:01.959373 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:01.959353    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:08:08.186354 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:08.186316    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 22:08:08.320635 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:08.320595    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Oct 30 22:08:08.321034 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:08.321012    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:08:11.303288 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:08:11.303253    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:08:18.185922 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:18.185878    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 22:08:18.321138 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:18.321096    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Oct 30 22:08:18.321533 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:18.321510    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:08:20.881397 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:20.881358    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 22:08:28.326924 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:28.321444    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Oct 30 22:08:28.327528 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:28.327308    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:08:30.881893 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:30.881858    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:08:38.320755 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:38.320717    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Oct 30 22:08:38.321284 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:38.321167    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:08:41.308190 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:08:41.303119    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:08:48.320624 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:48.320541    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Oct 30 22:08:48.321043 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:48.321000    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:08:50.212807 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.212771    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:50.468756 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.468666    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:50.683807 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.683773    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"]
Oct 30 22:08:50.687354 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.687326    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:08:50.740611 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.740536    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-secondary-0052a3-kube-rbac-proxy-sar-config\""
Oct 30 22:08:50.758922 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.758899    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4z85\" (UniqueName: \"kubernetes.io/projected/f2b68112-2347-430b-9d9f-f6e1a53e870a-kube-api-access-g4z85\") pod \"isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") " pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:08:50.759056 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.758964    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2b68112-2347-430b-9d9f-f6e1a53e870a-proxy-tls\") pod \"isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") " pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:08:50.759056 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.759042    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2b68112-2347-430b-9d9f-f6e1a53e870a-cabundle-cert\") pod \"isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") " pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:08:50.759164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.759092    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2b68112-2347-430b-9d9f-f6e1a53e870a-kserve-provision-location\") pod \"isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") " pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:08:50.759218 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.759183    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2b68112-2347-430b-9d9f-f6e1a53e870a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") " pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:08:50.790819 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.790791    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"invalid-s3-secret-0052a3\""
Oct 30 22:08:50.790987 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.790826    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"invalid-s3-sa-0052a3-dockercfg-l7nw6\""
Oct 30 22:08:50.790987 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.790934    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-secondary-0052a3-predictor-serving-cert\""
Oct 30 22:08:50.792613 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.792593    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"]
Oct 30 22:08:50.859854 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.859827    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2b68112-2347-430b-9d9f-f6e1a53e870a-proxy-tls\") pod \"isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") " pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:08:50.860006 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.859858    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2b68112-2347-430b-9d9f-f6e1a53e870a-cabundle-cert\") pod \"isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") " pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:08:50.860006 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.859890    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2b68112-2347-430b-9d9f-f6e1a53e870a-kserve-provision-location\") pod \"isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") " pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:08:50.860006 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.859927    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2b68112-2347-430b-9d9f-f6e1a53e870a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") " pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:08:50.860006 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.859982    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4z85\" (UniqueName: \"kubernetes.io/projected/f2b68112-2347-430b-9d9f-f6e1a53e870a-kube-api-access-g4z85\") pod \"isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") " pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:08:50.860313 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.860282    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2b68112-2347-430b-9d9f-f6e1a53e870a-kserve-provision-location\") pod \"isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") " pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:08:50.860634 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.860618    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2b68112-2347-430b-9d9f-f6e1a53e870a-cabundle-cert\") pod \"isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") " pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:08:50.860704 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.860651    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2b68112-2347-430b-9d9f-f6e1a53e870a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") " pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:08:50.862425 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.862404    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2b68112-2347-430b-9d9f-f6e1a53e870a-proxy-tls\") pod \"isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") " pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:08:50.918250 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:50.918227    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4z85\" (UniqueName: \"kubernetes.io/projected/f2b68112-2347-430b-9d9f-f6e1a53e870a-kube-api-access-g4z85\") pod \"isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") " pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:08:51.004591 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:51.004513    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:08:51.429360 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:51.429315    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"]
Oct 30 22:08:51.432487 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:08:51.432458    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b68112_2347_430b_9d9f_f6e1a53e870a.slice/crio-5fb2d122cac2df9cfdf95183b7ed0d9493932593045d2e6a3478db90da00f269 WatchSource:0}: Error finding container 5fb2d122cac2df9cfdf95183b7ed0d9493932593045d2e6a3478db90da00f269: Status 404 returned error can't find the container with id 5fb2d122cac2df9cfdf95183b7ed0d9493932593045d2e6a3478db90da00f269
Oct 30 22:08:51.601878 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:51.601836    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg" event={"ID":"f2b68112-2347-430b-9d9f-f6e1a53e870a","Type":"ContainerStarted","Data":"4f67a6686696e158eb5dfff34eb8b2aa04cfc73bdae5e562a18852a6a64b5bea"}
Oct 30 22:08:51.602065 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:51.601915    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg" event={"ID":"f2b68112-2347-430b-9d9f-f6e1a53e870a","Type":"ContainerStarted","Data":"5fb2d122cac2df9cfdf95183b7ed0d9493932593045d2e6a3478db90da00f269"}
Oct 30 22:08:56.621933 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:56.621897    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:57.628856 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:57.628829    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg_f2b68112-2347-430b-9d9f-f6e1a53e870a/storage-initializer/0.log"
Oct 30 22:08:57.629347 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:57.628863    4473 generic.go:356] "Generic (PLEG): container finished" podID="f2b68112-2347-430b-9d9f-f6e1a53e870a" containerID="4f67a6686696e158eb5dfff34eb8b2aa04cfc73bdae5e562a18852a6a64b5bea" exitCode=1
Oct 30 22:08:57.629347 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:57.628966    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg" event={"ID":"f2b68112-2347-430b-9d9f-f6e1a53e870a","Type":"ContainerDied","Data":"4f67a6686696e158eb5dfff34eb8b2aa04cfc73bdae5e562a18852a6a64b5bea"}
Oct 30 22:08:58.322121 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:58.322090    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:08:58.322350 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:58.322334    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:08:58.640109 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:58.640030    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg_f2b68112-2347-430b-9d9f-f6e1a53e870a/storage-initializer/0.log"
Oct 30 22:08:58.640606 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:08:58.640148    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg" event={"ID":"f2b68112-2347-430b-9d9f-f6e1a53e870a","Type":"ContainerStarted","Data":"5b2e1b39067dd090492c311bfe122c374b808d6a037fe8c59c0acfc1b7ed49ba"}
Oct 30 22:09:03.661640 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:03.661616    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg_f2b68112-2347-430b-9d9f-f6e1a53e870a/storage-initializer/1.log"
Oct 30 22:09:03.662056 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:03.662034    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg_f2b68112-2347-430b-9d9f-f6e1a53e870a/storage-initializer/0.log"
Oct 30 22:09:03.662114 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:03.662067    4473 generic.go:356] "Generic (PLEG): container finished" podID="f2b68112-2347-430b-9d9f-f6e1a53e870a" containerID="5b2e1b39067dd090492c311bfe122c374b808d6a037fe8c59c0acfc1b7ed49ba" exitCode=1
Oct 30 22:09:03.662152 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:03.662125    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg" event={"ID":"f2b68112-2347-430b-9d9f-f6e1a53e870a","Type":"ContainerDied","Data":"5b2e1b39067dd090492c311bfe122c374b808d6a037fe8c59c0acfc1b7ed49ba"}
Oct 30 22:09:03.662189 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:03.662182    4473 scope.go:122] "RemoveContainer" containerID="4f67a6686696e158eb5dfff34eb8b2aa04cfc73bdae5e562a18852a6a64b5bea"
Oct 30 22:09:03.662630 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:03.662611    4473 scope.go:122] "RemoveContainer" containerID="4f67a6686696e158eb5dfff34eb8b2aa04cfc73bdae5e562a18852a6a64b5bea"
Oct 30 22:09:03.673275 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:09:03.673254    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg_e2e-gw1-7476d02544(f2b68112-2347-430b-9d9f-f6e1a53e870a)\"" pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg" podUID="f2b68112-2347-430b-9d9f-f6e1a53e870a"
Oct 30 22:09:03.673664 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:09:03.673635    4473 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg_e2e-gw1-7476d02544_f2b68112-2347-430b-9d9f-f6e1a53e870a_0 in pod sandbox 5fb2d122cac2df9cfdf95183b7ed0d9493932593045d2e6a3478db90da00f269 from index: no such id: '4f67a6686696e158eb5dfff34eb8b2aa04cfc73bdae5e562a18852a6a64b5bea'" containerID="4f67a6686696e158eb5dfff34eb8b2aa04cfc73bdae5e562a18852a6a64b5bea"
Oct 30 22:09:03.673760 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:03.673672    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f67a6686696e158eb5dfff34eb8b2aa04cfc73bdae5e562a18852a6a64b5bea"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg_e2e-gw1-7476d02544_f2b68112-2347-430b-9d9f-f6e1a53e870a_0 in pod sandbox 5fb2d122cac2df9cfdf95183b7ed0d9493932593045d2e6a3478db90da00f269 from index: no such id: '4f67a6686696e158eb5dfff34eb8b2aa04cfc73bdae5e562a18852a6a64b5bea'"
Oct 30 22:09:04.667641 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:04.667614    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg_f2b68112-2347-430b-9d9f-f6e1a53e870a/storage-initializer/1.log"
Oct 30 22:09:10.841477 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:10.841430    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"]
Oct 30 22:09:10.842096 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:10.841868    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kserve-container" containerID="cri-o://729980ed09a58e75b916064e4d43db69b5cb0bd33a43de07be08d33dd831864c" gracePeriod=30
Oct 30 22:09:10.842210 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:10.842175    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kube-rbac-proxy" containerID="cri-o://e0cb459be318d5d3b123d146fe291288266b10b5621d17bac1bff5e23fd7841d" gracePeriod=30
Oct 30 22:09:10.881816 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:10.881777    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 22:09:10.905127 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:10.905096    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:11.304095 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:09:11.304063    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:09:11.358113 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:11.358084    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"]
Oct 30 22:09:11.358391 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:11.358368    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kserve-container" containerID="cri-o://e329ed3988a516737d9d3749c5426c6207435998444628c38276e5fe8338289b" gracePeriod=30
Oct 30 22:09:11.358482 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:11.358454    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="agent" containerID="cri-o://ff901a7099a1ec48733fb834cdb168dcac5d9a738c2d88547b3e72e1f17eca85" gracePeriod=30
Oct 30 22:09:11.358548 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:11.358492    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kube-rbac-proxy" containerID="cri-o://959e027195ba0d43ecda0574d81a2fa74bd97d873ef50f84beebccf0bbf2c480" gracePeriod=30
Oct 30 22:09:11.396417 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:11.396389    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:11.701261 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:11.701226    4473 generic.go:356] "Generic (PLEG): container finished" podID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerID="e0cb459be318d5d3b123d146fe291288266b10b5621d17bac1bff5e23fd7841d" exitCode=0
Oct 30 22:09:11.701455 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:11.701298    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" event={"ID":"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d","Type":"ContainerDied","Data":"e0cb459be318d5d3b123d146fe291288266b10b5621d17bac1bff5e23fd7841d"}
Oct 30 22:09:11.703732 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:11.703706    4473 generic.go:356] "Generic (PLEG): container finished" podID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerID="959e027195ba0d43ecda0574d81a2fa74bd97d873ef50f84beebccf0bbf2c480" exitCode=0
Oct 30 22:09:11.703872 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:11.703779    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" event={"ID":"fa3e75b0-ddd8-481d-9bca-2e10f0b28061","Type":"ContainerDied","Data":"959e027195ba0d43ecda0574d81a2fa74bd97d873ef50f84beebccf0bbf2c480"}
Oct 30 22:09:11.890502 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:11.890466    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"]
Oct 30 22:09:11.920753 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:11.920721    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:12.016943 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.016923    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg_f2b68112-2347-430b-9d9f-f6e1a53e870a/storage-initializer/1.log"
Oct 30 22:09:12.019185 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.019170    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:09:12.040227 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.040201    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b68112-2347-430b-9d9f-f6e1a53e870a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2b68112-2347-430b-9d9f-f6e1a53e870a-kube-rbac-proxy-sar-config\") pod \"f2b68112-2347-430b-9d9f-f6e1a53e870a\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") "
Oct 30 22:09:12.040325 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.040241    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b68112-2347-430b-9d9f-f6e1a53e870a-proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2b68112-2347-430b-9d9f-f6e1a53e870a-proxy-tls\") pod \"f2b68112-2347-430b-9d9f-f6e1a53e870a\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") "
Oct 30 22:09:12.040325 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.040262    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b68112-2347-430b-9d9f-f6e1a53e870a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2b68112-2347-430b-9d9f-f6e1a53e870a-cabundle-cert\") pod \"f2b68112-2347-430b-9d9f-f6e1a53e870a\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") "
Oct 30 22:09:12.040415 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.040344    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2b68112-2347-430b-9d9f-f6e1a53e870a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2b68112-2347-430b-9d9f-f6e1a53e870a-kserve-provision-location\") pod \"f2b68112-2347-430b-9d9f-f6e1a53e870a\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") "
Oct 30 22:09:12.040489 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.040470    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b68112-2347-430b-9d9f-f6e1a53e870a-kube-api-access-g4z85\" (UniqueName: \"kubernetes.io/projected/f2b68112-2347-430b-9d9f-f6e1a53e870a-kube-api-access-g4z85\") pod \"f2b68112-2347-430b-9d9f-f6e1a53e870a\" (UID: \"f2b68112-2347-430b-9d9f-f6e1a53e870a\") "
Oct 30 22:09:12.040587 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.040550    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b68112-2347-430b-9d9f-f6e1a53e870a-kube-rbac-proxy-sar-config" pod "f2b68112-2347-430b-9d9f-f6e1a53e870a" (UID: "f2b68112-2347-430b-9d9f-f6e1a53e870a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:12.040641 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.040554    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b68112-2347-430b-9d9f-f6e1a53e870a-kserve-provision-location" pod "f2b68112-2347-430b-9d9f-f6e1a53e870a" (UID: "f2b68112-2347-430b-9d9f-f6e1a53e870a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:12.040692 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.040632    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b68112-2347-430b-9d9f-f6e1a53e870a-cabundle-cert" pod "f2b68112-2347-430b-9d9f-f6e1a53e870a" (UID: "f2b68112-2347-430b-9d9f-f6e1a53e870a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:12.040755 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.040739    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2b68112-2347-430b-9d9f-f6e1a53e870a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:12.040807 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.040765    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2b68112-2347-430b-9d9f-f6e1a53e870a-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:12.040807 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.040782    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2b68112-2347-430b-9d9f-f6e1a53e870a-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:12.042534 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.042516    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b68112-2347-430b-9d9f-f6e1a53e870a-proxy-tls" pod "f2b68112-2347-430b-9d9f-f6e1a53e870a" (UID: "f2b68112-2347-430b-9d9f-f6e1a53e870a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:12.042731 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.042705    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b68112-2347-430b-9d9f-f6e1a53e870a-kube-api-access-g4z85" pod "f2b68112-2347-430b-9d9f-f6e1a53e870a" (UID: "f2b68112-2347-430b-9d9f-f6e1a53e870a"). InnerVolumeSpecName "kube-api-access-g4z85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:12.141706 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.141672    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4z85\" (UniqueName: \"kubernetes.io/projected/f2b68112-2347-430b-9d9f-f6e1a53e870a-kube-api-access-g4z85\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:12.141706 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.141699    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2b68112-2347-430b-9d9f-f6e1a53e870a-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:12.709036 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.709009    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg_f2b68112-2347-430b-9d9f-f6e1a53e870a/storage-initializer/1.log"
Oct 30 22:09:12.709207 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.709070    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg" event={"ID":"f2b68112-2347-430b-9d9f-f6e1a53e870a","Type":"ContainerDied","Data":"5fb2d122cac2df9cfdf95183b7ed0d9493932593045d2e6a3478db90da00f269"}
Oct 30 22:09:12.709207 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.709105    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"
Oct 30 22:09:12.709207 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.709121    4473 scope.go:122] "RemoveContainer" containerID="5b2e1b39067dd090492c311bfe122c374b808d6a037fe8c59c0acfc1b7ed49ba"
Oct 30 22:09:12.995578 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:12.995498    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"]
Oct 30 22:09:13.045933 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:13.045901    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-secondary-0052a3-predictor-57dd7769b7-qw9sg"]
Oct 30 22:09:13.181137 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:13.181096    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Oct 30 22:09:13.316211 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:13.316118    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Oct 30 22:09:13.886237 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:13.886205    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b68112-2347-430b-9d9f-f6e1a53e870a" path="/var/lib/kubelet/pods/f2b68112-2347-430b-9d9f-f6e1a53e870a/volumes"
Oct 30 22:09:16.091010 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.090986    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:09:16.178177 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.178148    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kserve-provision-location\") pod \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") "
Oct 30 22:09:16.178177 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.178190    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kube-rbac-proxy-sar-config\") pod \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") "
Oct 30 22:09:16.178389 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.178216    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kube-api-access-9lcw8\" (UniqueName: \"kubernetes.io/projected/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kube-api-access-9lcw8\") pod \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") "
Oct 30 22:09:16.178389 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.178247    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-cabundle-cert\") pod \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") "
Oct 30 22:09:16.178389 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.178312    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-proxy-tls\") pod \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\" (UID: \"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d\") "
Oct 30 22:09:16.178533 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.178440    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kserve-provision-location" pod "e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" (UID: "e85912ff-a3f2-4c2c-9d92-2883e01d5b8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:16.178610 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.178559    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.178670 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.178610    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kube-rbac-proxy-sar-config" pod "e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" (UID: "e85912ff-a3f2-4c2c-9d92-2883e01d5b8d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:16.178670 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.178646    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-cabundle-cert" pod "e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" (UID: "e85912ff-a3f2-4c2c-9d92-2883e01d5b8d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:16.180455 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.180432    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kube-api-access-9lcw8" pod "e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" (UID: "e85912ff-a3f2-4c2c-9d92-2883e01d5b8d"). InnerVolumeSpecName "kube-api-access-9lcw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:16.180551 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.180493    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-proxy-tls" pod "e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" (UID: "e85912ff-a3f2-4c2c-9d92-2883e01d5b8d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:16.279740 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.279710    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.279740 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.279737    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.279927 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.279751    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.279927 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.279764    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lcw8\" (UniqueName: \"kubernetes.io/projected/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d-kube-api-access-9lcw8\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.729541 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.729455    4473 generic.go:356] "Generic (PLEG): container finished" podID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerID="729980ed09a58e75b916064e4d43db69b5cb0bd33a43de07be08d33dd831864c" exitCode=0
Oct 30 22:09:16.729541 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.729514    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"
Oct 30 22:09:16.729789 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.729549    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" event={"ID":"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d","Type":"ContainerDied","Data":"729980ed09a58e75b916064e4d43db69b5cb0bd33a43de07be08d33dd831864c"}
Oct 30 22:09:16.729789 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.729622    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27" event={"ID":"e85912ff-a3f2-4c2c-9d92-2883e01d5b8d","Type":"ContainerDied","Data":"9d33c199cdcecc7b091326052a0d43c5411effd8e320e60e0c4e556fe698f8a4"}
Oct 30 22:09:16.729789 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.729660    4473 scope.go:122] "RemoveContainer" containerID="e0cb459be318d5d3b123d146fe291288266b10b5621d17bac1bff5e23fd7841d"
Oct 30 22:09:16.732481 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.732454    4473 generic.go:356] "Generic (PLEG): container finished" podID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerID="e329ed3988a516737d9d3749c5426c6207435998444628c38276e5fe8338289b" exitCode=0
Oct 30 22:09:16.732584 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.732529    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" event={"ID":"fa3e75b0-ddd8-481d-9bca-2e10f0b28061","Type":"ContainerDied","Data":"e329ed3988a516737d9d3749c5426c6207435998444628c38276e5fe8338289b"}
Oct 30 22:09:16.738366 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.738350    4473 scope.go:122] "RemoveContainer" containerID="729980ed09a58e75b916064e4d43db69b5cb0bd33a43de07be08d33dd831864c"
Oct 30 22:09:16.745803 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.745788    4473 scope.go:122] "RemoveContainer" containerID="d28b0d8081196df1b9d4a4d8517cc3161bc748c76c23cf9271737a73a813dee1"
Oct 30 22:09:16.753085 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.753065    4473 scope.go:122] "RemoveContainer" containerID="e0cb459be318d5d3b123d146fe291288266b10b5621d17bac1bff5e23fd7841d"
Oct 30 22:09:16.753366 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:09:16.753348    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0cb459be318d5d3b123d146fe291288266b10b5621d17bac1bff5e23fd7841d\": container with ID starting with e0cb459be318d5d3b123d146fe291288266b10b5621d17bac1bff5e23fd7841d not found: ID does not exist" containerID="e0cb459be318d5d3b123d146fe291288266b10b5621d17bac1bff5e23fd7841d"
Oct 30 22:09:16.753415 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.753376    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0cb459be318d5d3b123d146fe291288266b10b5621d17bac1bff5e23fd7841d"} err="failed to get container status \"e0cb459be318d5d3b123d146fe291288266b10b5621d17bac1bff5e23fd7841d\": rpc error: code = NotFound desc = could not find container \"e0cb459be318d5d3b123d146fe291288266b10b5621d17bac1bff5e23fd7841d\": container with ID starting with e0cb459be318d5d3b123d146fe291288266b10b5621d17bac1bff5e23fd7841d not found: ID does not exist"
Oct 30 22:09:16.753415 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.753394    4473 scope.go:122] "RemoveContainer" containerID="729980ed09a58e75b916064e4d43db69b5cb0bd33a43de07be08d33dd831864c"
Oct 30 22:09:16.753599 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:09:16.753580    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729980ed09a58e75b916064e4d43db69b5cb0bd33a43de07be08d33dd831864c\": container with ID starting with 729980ed09a58e75b916064e4d43db69b5cb0bd33a43de07be08d33dd831864c not found: ID does not exist" containerID="729980ed09a58e75b916064e4d43db69b5cb0bd33a43de07be08d33dd831864c"
Oct 30 22:09:16.753677 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.753610    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729980ed09a58e75b916064e4d43db69b5cb0bd33a43de07be08d33dd831864c"} err="failed to get container status \"729980ed09a58e75b916064e4d43db69b5cb0bd33a43de07be08d33dd831864c\": rpc error: code = NotFound desc = could not find container \"729980ed09a58e75b916064e4d43db69b5cb0bd33a43de07be08d33dd831864c\": container with ID starting with 729980ed09a58e75b916064e4d43db69b5cb0bd33a43de07be08d33dd831864c not found: ID does not exist"
Oct 30 22:09:16.753677 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.753631    4473 scope.go:122] "RemoveContainer" containerID="d28b0d8081196df1b9d4a4d8517cc3161bc748c76c23cf9271737a73a813dee1"
Oct 30 22:09:16.753869 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:09:16.753848    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d28b0d8081196df1b9d4a4d8517cc3161bc748c76c23cf9271737a73a813dee1\": container with ID starting with d28b0d8081196df1b9d4a4d8517cc3161bc748c76c23cf9271737a73a813dee1 not found: ID does not exist" containerID="d28b0d8081196df1b9d4a4d8517cc3161bc748c76c23cf9271737a73a813dee1"
Oct 30 22:09:16.753919 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.753874    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d28b0d8081196df1b9d4a4d8517cc3161bc748c76c23cf9271737a73a813dee1"} err="failed to get container status \"d28b0d8081196df1b9d4a4d8517cc3161bc748c76c23cf9271737a73a813dee1\": rpc error: code = NotFound desc = could not find container \"d28b0d8081196df1b9d4a4d8517cc3161bc748c76c23cf9271737a73a813dee1\": container with ID starting with d28b0d8081196df1b9d4a4d8517cc3161bc748c76c23cf9271737a73a813dee1 not found: ID does not exist"
Oct 30 22:09:16.948064 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.948028    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"]
Oct 30 22:09:16.951550 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:16.951529    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-primary-0052a3-predictor-78b6456644-g8m27"]
Oct 30 22:09:17.886065 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:17.886033    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" path="/var/lib/kubelet/pods/e85912ff-a3f2-4c2c-9d92-2883e01d5b8d/volumes"
Oct 30 22:09:18.316587 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:18.316547    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Oct 30 22:09:18.320716 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:18.320678    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Oct 30 22:09:18.321018 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:18.320994    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:09:20.325552 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.325508    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:20.637202 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.637115    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:20.822568 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.822534    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"]
Oct 30 22:09:20.822876 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.822864    4473 cpu_manager.go:414] "containerMap: removing container" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kube-rbac-proxy"
Oct 30 22:09:20.822915 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.822880    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kube-rbac-proxy"
Oct 30 22:09:20.822915 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.822909    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f2b68112-2347-430b-9d9f-f6e1a53e870a" containerName="storage-initializer"
Oct 30 22:09:20.822998 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.822915    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b68112-2347-430b-9d9f-f6e1a53e870a" containerName="storage-initializer"
Oct 30 22:09:20.822998 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.822926    4473 cpu_manager.go:414] "containerMap: removing container" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kserve-container"
Oct 30 22:09:20.822998 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.822931    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kserve-container"
Oct 30 22:09:20.822998 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.822941    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f2b68112-2347-430b-9d9f-f6e1a53e870a" containerName="storage-initializer"
Oct 30 22:09:20.822998 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.822946    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b68112-2347-430b-9d9f-f6e1a53e870a" containerName="storage-initializer"
Oct 30 22:09:20.822998 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.822978    4473 cpu_manager.go:414] "containerMap: removing container" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="storage-initializer"
Oct 30 22:09:20.822998 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.822983    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="storage-initializer"
Oct 30 22:09:20.823209 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.823035    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b68112-2347-430b-9d9f-f6e1a53e870a" containerName="storage-initializer"
Oct 30 22:09:20.823209 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.823047    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kserve-container"
Oct 30 22:09:20.823209 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.823055    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b68112-2347-430b-9d9f-f6e1a53e870a" containerName="storage-initializer"
Oct 30 22:09:20.823209 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.823064    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e85912ff-a3f2-4c2c-9d92-2883e01d5b8d" containerName="kube-rbac-proxy"
Oct 30 22:09:20.827710 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.827690    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:20.881194 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.881164    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:09:20.881194 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.881184    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-init-fail-1df802-kube-rbac-proxy-sar-config\""
Oct 30 22:09:20.921169 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.921142    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82sm\" (UniqueName: \"kubernetes.io/projected/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kube-api-access-g82sm\") pod \"isvc-init-fail-1df802-predictor-7784db59-rlbkb\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") " pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:20.921352 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.921176    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kserve-provision-location\") pod \"isvc-init-fail-1df802-predictor-7784db59-rlbkb\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") " pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:20.921352 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.921199    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-1df802-predictor-7784db59-rlbkb\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") " pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:20.921352 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.921338    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/013a6aa1-9618-485e-b6b6-acf4a36be8b5-cabundle-cert\") pod \"isvc-init-fail-1df802-predictor-7784db59-rlbkb\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") " pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:20.921544 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.921374    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/013a6aa1-9618-485e-b6b6-acf4a36be8b5-proxy-tls\") pod \"isvc-init-fail-1df802-predictor-7784db59-rlbkb\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") " pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:20.931512 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.931485    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:09:20.931512 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.931500    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:09:20.931709 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.931501    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-init-fail-1df802-predictor-serving-cert\""
Oct 30 22:09:20.931709 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.931524    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"fail-s3-secret-1df802\""
Oct 30 22:09:20.931709 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.931663    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"fail-s3-sa-1df802-dockercfg-kn95c\""
Oct 30 22:09:20.933197 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:20.933173    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"]
Oct 30 22:09:21.022746 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:21.022710    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/013a6aa1-9618-485e-b6b6-acf4a36be8b5-cabundle-cert\") pod \"isvc-init-fail-1df802-predictor-7784db59-rlbkb\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") " pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:21.022924 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:21.022755    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/013a6aa1-9618-485e-b6b6-acf4a36be8b5-proxy-tls\") pod \"isvc-init-fail-1df802-predictor-7784db59-rlbkb\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") " pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:21.022924 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:21.022821    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82sm\" (UniqueName: \"kubernetes.io/projected/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kube-api-access-g82sm\") pod \"isvc-init-fail-1df802-predictor-7784db59-rlbkb\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") " pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:21.022924 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:21.022847    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kserve-provision-location\") pod \"isvc-init-fail-1df802-predictor-7784db59-rlbkb\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") " pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:21.022924 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:21.022880    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-1df802-predictor-7784db59-rlbkb\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") " pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:21.023280 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:21.023259    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kserve-provision-location\") pod \"isvc-init-fail-1df802-predictor-7784db59-rlbkb\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") " pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:21.023596 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:21.023574    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/013a6aa1-9618-485e-b6b6-acf4a36be8b5-cabundle-cert\") pod \"isvc-init-fail-1df802-predictor-7784db59-rlbkb\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") " pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:21.023641 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:21.023574    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-1df802-predictor-7784db59-rlbkb\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") " pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:21.025387 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:21.025370    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/013a6aa1-9618-485e-b6b6-acf4a36be8b5-proxy-tls\") pod \"isvc-init-fail-1df802-predictor-7784db59-rlbkb\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") " pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:21.080645 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:21.080612    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82sm\" (UniqueName: \"kubernetes.io/projected/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kube-api-access-g82sm\") pod \"isvc-init-fail-1df802-predictor-7784db59-rlbkb\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") " pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:21.144755 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:21.144718    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:21.566635 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:21.566551    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"]
Oct 30 22:09:21.568968 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:09:21.568920    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013a6aa1_9618_485e_b6b6_acf4a36be8b5.slice/crio-20a4218f9dc5a8ef1cad4c490240f7bcad3543c89b2d0c889102300a1b58c78f WatchSource:0}: Error finding container 20a4218f9dc5a8ef1cad4c490240f7bcad3543c89b2d0c889102300a1b58c78f: Status 404 returned error can't find the container with id 20a4218f9dc5a8ef1cad4c490240f7bcad3543c89b2d0c889102300a1b58c78f
Oct 30 22:09:21.759718 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:21.756734    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb" event={"ID":"013a6aa1-9618-485e-b6b6-acf4a36be8b5","Type":"ContainerStarted","Data":"0950f32bc79ecdf152085e8f470db518e990d64c7f3f08a4951516599e428c76"}
Oct 30 22:09:21.759718 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:21.756896    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb" event={"ID":"013a6aa1-9618-485e-b6b6-acf4a36be8b5","Type":"ContainerStarted","Data":"20a4218f9dc5a8ef1cad4c490240f7bcad3543c89b2d0c889102300a1b58c78f"}
Oct 30 22:09:23.316920 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:23.316880    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Oct 30 22:09:23.317339 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:23.317033    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:09:26.776522 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:26.776449    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-init-fail-1df802-predictor-7784db59-rlbkb_013a6aa1-9618-485e-b6b6-acf4a36be8b5/storage-initializer/0.log"
Oct 30 22:09:26.776522 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:26.776485    4473 generic.go:356] "Generic (PLEG): container finished" podID="013a6aa1-9618-485e-b6b6-acf4a36be8b5" containerID="0950f32bc79ecdf152085e8f470db518e990d64c7f3f08a4951516599e428c76" exitCode=1
Oct 30 22:09:26.776912 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:26.776533    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb" event={"ID":"013a6aa1-9618-485e-b6b6-acf4a36be8b5","Type":"ContainerDied","Data":"0950f32bc79ecdf152085e8f470db518e990d64c7f3f08a4951516599e428c76"}
Oct 30 22:09:27.782995 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:27.782946    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-init-fail-1df802-predictor-7784db59-rlbkb_013a6aa1-9618-485e-b6b6-acf4a36be8b5/storage-initializer/0.log"
Oct 30 22:09:27.783400 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:27.783076    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb" event={"ID":"013a6aa1-9618-485e-b6b6-acf4a36be8b5","Type":"ContainerStarted","Data":"310d78c7f7e7d2e3d61bea1b2292e64b987cffd6c5768ded75d52dc4103585a4"}
Oct 30 22:09:28.316014 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:28.315979    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Oct 30 22:09:28.321101 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:28.321078    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Oct 30 22:09:28.321510 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:28.321488    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:09:30.562046 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:30.562015    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"]
Oct 30 22:09:30.562469 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:30.562219    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb" podUID="013a6aa1-9618-485e-b6b6-acf4a36be8b5" containerName="storage-initializer" containerID="cri-o://310d78c7f7e7d2e3d61bea1b2292e64b987cffd6c5768ded75d52dc4103585a4" gracePeriod=30
Oct 30 22:09:30.598495 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:30.598460    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:31.801980 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:31.801934    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-init-fail-1df802-predictor-7784db59-rlbkb_013a6aa1-9618-485e-b6b6-acf4a36be8b5/storage-initializer/1.log"
Oct 30 22:09:31.802428 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:31.802349    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-init-fail-1df802-predictor-7784db59-rlbkb_013a6aa1-9618-485e-b6b6-acf4a36be8b5/storage-initializer/0.log"
Oct 30 22:09:31.802428 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:31.802380    4473 generic.go:356] "Generic (PLEG): container finished" podID="013a6aa1-9618-485e-b6b6-acf4a36be8b5" containerID="310d78c7f7e7d2e3d61bea1b2292e64b987cffd6c5768ded75d52dc4103585a4" exitCode=1
Oct 30 22:09:31.802538 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:31.802457    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb" event={"ID":"013a6aa1-9618-485e-b6b6-acf4a36be8b5","Type":"ContainerDied","Data":"310d78c7f7e7d2e3d61bea1b2292e64b987cffd6c5768ded75d52dc4103585a4"}
Oct 30 22:09:31.802600 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:31.802544    4473 scope.go:122] "RemoveContainer" containerID="0950f32bc79ecdf152085e8f470db518e990d64c7f3f08a4951516599e428c76"
Oct 30 22:09:31.899307 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:31.899286    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-init-fail-1df802-predictor-7784db59-rlbkb_013a6aa1-9618-485e-b6b6-acf4a36be8b5/storage-initializer/1.log"
Oct 30 22:09:31.902709 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:31.902694    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:32.005808 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.005724    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kube-rbac-proxy-sar-config\") pod \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") "
Oct 30 22:09:32.005808 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.005758    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/013a6aa1-9618-485e-b6b6-acf4a36be8b5-proxy-tls\" (UniqueName: \"kubernetes.io/secret/013a6aa1-9618-485e-b6b6-acf4a36be8b5-proxy-tls\") pod \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") "
Oct 30 22:09:32.005808 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.005788    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013a6aa1-9618-485e-b6b6-acf4a36be8b5-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/013a6aa1-9618-485e-b6b6-acf4a36be8b5-cabundle-cert\") pod \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") "
Oct 30 22:09:32.006112 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.005850    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kube-api-access-g82sm\" (UniqueName: \"kubernetes.io/projected/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kube-api-access-g82sm\") pod \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") "
Oct 30 22:09:32.006112 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.005881    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kserve-provision-location\") pod \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\" (UID: \"013a6aa1-9618-485e-b6b6-acf4a36be8b5\") "
Oct 30 22:09:32.006286 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.006249    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013a6aa1-9618-485e-b6b6-acf4a36be8b5-cabundle-cert" pod "013a6aa1-9618-485e-b6b6-acf4a36be8b5" (UID: "013a6aa1-9618-485e-b6b6-acf4a36be8b5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:32.006286 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.006262    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kserve-provision-location" pod "013a6aa1-9618-485e-b6b6-acf4a36be8b5" (UID: "013a6aa1-9618-485e-b6b6-acf4a36be8b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:32.006441 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.006249    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kube-rbac-proxy-sar-config" pod "013a6aa1-9618-485e-b6b6-acf4a36be8b5" (UID: "013a6aa1-9618-485e-b6b6-acf4a36be8b5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:32.008164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.008128    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kube-api-access-g82sm" pod "013a6aa1-9618-485e-b6b6-acf4a36be8b5" (UID: "013a6aa1-9618-485e-b6b6-acf4a36be8b5"). InnerVolumeSpecName "kube-api-access-g82sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:32.008249 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.008162    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/013a6aa1-9618-485e-b6b6-acf4a36be8b5-proxy-tls" pod "013a6aa1-9618-485e-b6b6-acf4a36be8b5" (UID: "013a6aa1-9618-485e-b6b6-acf4a36be8b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:32.107092 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.107060    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:32.107092 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.107088    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/013a6aa1-9618-485e-b6b6-acf4a36be8b5-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:32.107092 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.107100    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/013a6aa1-9618-485e-b6b6-acf4a36be8b5-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:32.107341 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.107109    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g82sm\" (UniqueName: \"kubernetes.io/projected/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kube-api-access-g82sm\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:32.107341 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.107118    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/013a6aa1-9618-485e-b6b6-acf4a36be8b5-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:32.807556 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.807529    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-init-fail-1df802-predictor-7784db59-rlbkb_013a6aa1-9618-485e-b6b6-acf4a36be8b5/storage-initializer/1.log"
Oct 30 22:09:32.807930 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.807643    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"
Oct 30 22:09:32.807930 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.807636    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb" event={"ID":"013a6aa1-9618-485e-b6b6-acf4a36be8b5","Type":"ContainerDied","Data":"20a4218f9dc5a8ef1cad4c490240f7bcad3543c89b2d0c889102300a1b58c78f"}
Oct 30 22:09:32.807930 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:32.807790    4473 scope.go:122] "RemoveContainer" containerID="310d78c7f7e7d2e3d61bea1b2292e64b987cffd6c5768ded75d52dc4103585a4"
Oct 30 22:09:33.092713 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:33.092632    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"]
Oct 30 22:09:33.096271 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:33.096243    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-init-fail-1df802-predictor-7784db59-rlbkb"]
Oct 30 22:09:33.316734 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:33.316697    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Oct 30 22:09:33.885615 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:33.885579    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="013a6aa1-9618-485e-b6b6-acf4a36be8b5" path="/var/lib/kubelet/pods/013a6aa1-9618-485e-b6b6-acf4a36be8b5/volumes"
Oct 30 22:09:36.277187 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.277149    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:36.764518 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.764479    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:36.768309 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.768288    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"]
Oct 30 22:09:36.768605 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.768593    4473 cpu_manager.go:414] "containerMap: removing container" podUID="013a6aa1-9618-485e-b6b6-acf4a36be8b5" containerName="storage-initializer"
Oct 30 22:09:36.768657 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.768607    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013a6aa1-9618-485e-b6b6-acf4a36be8b5" containerName="storage-initializer"
Oct 30 22:09:36.768695 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.768676    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="013a6aa1-9618-485e-b6b6-acf4a36be8b5" containerName="storage-initializer"
Oct 30 22:09:36.768763 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.768753    4473 cpu_manager.go:414] "containerMap: removing container" podUID="013a6aa1-9618-485e-b6b6-acf4a36be8b5" containerName="storage-initializer"
Oct 30 22:09:36.768763 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.768762    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013a6aa1-9618-485e-b6b6-acf4a36be8b5" containerName="storage-initializer"
Oct 30 22:09:36.768832 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.768816    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="013a6aa1-9618-485e-b6b6-acf4a36be8b5" containerName="storage-initializer"
Oct 30 22:09:36.773259 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.773243    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:09:36.826788 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.826762    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:09:36.826788 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.826772    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Oct 30 22:09:36.826788 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.826773    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:09:36.827048 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.826812    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:09:36.846187 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.846155    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d96451a-9e25-46de-8dc9-7dc91a5a8120-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:09:36.846295 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.846212    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndnsf\" (UniqueName: \"kubernetes.io/projected/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kube-api-access-ndnsf\") pod \"isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:09:36.846346 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.846324    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d96451a-9e25-46de-8dc9-7dc91a5a8120-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:09:36.846380 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.846363    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:09:36.846426 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.846382    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:09:36.877075 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.877043    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:09:36.877245 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.877058    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:09:36.877245 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.877096    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Oct 30 22:09:36.878613 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.878594    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"]
Oct 30 22:09:36.946846 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.946819    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndnsf\" (UniqueName: \"kubernetes.io/projected/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kube-api-access-ndnsf\") pod \"isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:09:36.947031 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.946855    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d96451a-9e25-46de-8dc9-7dc91a5a8120-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:09:36.947031 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.946881    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:09:36.947031 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.946898    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:09:36.947031 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.946930    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d96451a-9e25-46de-8dc9-7dc91a5a8120-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:09:36.947380 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.947360    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:09:36.947555 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.947535    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d96451a-9e25-46de-8dc9-7dc91a5a8120-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:09:36.947623 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.947605    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:09:36.949502 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:36.949481    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d96451a-9e25-46de-8dc9-7dc91a5a8120-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:09:37.005131 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:37.005098    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndnsf\" (UniqueName: \"kubernetes.io/projected/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kube-api-access-ndnsf\") pod \"isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:09:37.090554 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:37.090468    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:09:37.514746 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:37.514711    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"]
Oct 30 22:09:37.552032 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:09:37.552003    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d96451a_9e25_46de_8dc9_7dc91a5a8120.slice/crio-d6ef99cb1c09cec279e16f32d184167181598c467315f93c9ca1b7b2bc0ff455 WatchSource:0}: Error finding container d6ef99cb1c09cec279e16f32d184167181598c467315f93c9ca1b7b2bc0ff455: Status 404 returned error can't find the container with id d6ef99cb1c09cec279e16f32d184167181598c467315f93c9ca1b7b2bc0ff455
Oct 30 22:09:37.832020 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:37.831917    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" event={"ID":"1d96451a-9e25-46de-8dc9-7dc91a5a8120","Type":"ContainerStarted","Data":"38b7436d5cc8e10a614405ee498f22240bf4115b092e4c94dca5004fb5b71f31"}
Oct 30 22:09:37.832020 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:37.831989    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" event={"ID":"1d96451a-9e25-46de-8dc9-7dc91a5a8120","Type":"ContainerStarted","Data":"d6ef99cb1c09cec279e16f32d184167181598c467315f93c9ca1b7b2bc0ff455"}
Oct 30 22:09:38.316497 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:38.316466    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Oct 30 22:09:38.320805 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:38.320770    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Oct 30 22:09:38.320923 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:38.320906    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:09:38.321102 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:38.321085    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:09:38.321185 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:38.321175    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:09:41.303451 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:09:41.303415    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:09:41.556917 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.556853    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:09:41.690058 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.690028    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kube-rbac-proxy-sar-config\") pod \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") "
Oct 30 22:09:41.690205 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.690086    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kserve-provision-location\") pod \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") "
Oct 30 22:09:41.690205 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.690133    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kube-api-access-5bdck\" (UniqueName: \"kubernetes.io/projected/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kube-api-access-5bdck\") pod \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") "
Oct 30 22:09:41.690205 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.690175    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-cabundle-cert\") pod \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") "
Oct 30 22:09:41.690351 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.690212    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-proxy-tls\") pod \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\" (UID: \"fa3e75b0-ddd8-481d-9bca-2e10f0b28061\") "
Oct 30 22:09:41.690463 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.690441    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kserve-provision-location" pod "fa3e75b0-ddd8-481d-9bca-2e10f0b28061" (UID: "fa3e75b0-ddd8-481d-9bca-2e10f0b28061"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:41.690526 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.690481    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kube-rbac-proxy-sar-config" pod "fa3e75b0-ddd8-481d-9bca-2e10f0b28061" (UID: "fa3e75b0-ddd8-481d-9bca-2e10f0b28061"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:41.690526 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.690519    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-cabundle-cert" pod "fa3e75b0-ddd8-481d-9bca-2e10f0b28061" (UID: "fa3e75b0-ddd8-481d-9bca-2e10f0b28061"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:41.692387 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.692366    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-proxy-tls" pod "fa3e75b0-ddd8-481d-9bca-2e10f0b28061" (UID: "fa3e75b0-ddd8-481d-9bca-2e10f0b28061"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:41.692588 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.692573    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kube-api-access-5bdck" pod "fa3e75b0-ddd8-481d-9bca-2e10f0b28061" (UID: "fa3e75b0-ddd8-481d-9bca-2e10f0b28061"). InnerVolumeSpecName "kube-api-access-5bdck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:41.791385 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.791349    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bdck\" (UniqueName: \"kubernetes.io/projected/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kube-api-access-5bdck\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:41.791385 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.791379    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:41.791385 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.791390    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:41.791621 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.791399    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:41.791621 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.791408    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa3e75b0-ddd8-481d-9bca-2e10f0b28061-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:41.849136 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.849049    4473 generic.go:356] "Generic (PLEG): container finished" podID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerID="38b7436d5cc8e10a614405ee498f22240bf4115b092e4c94dca5004fb5b71f31" exitCode=0
Oct 30 22:09:41.849136 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.849118    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" event={"ID":"1d96451a-9e25-46de-8dc9-7dc91a5a8120","Type":"ContainerDied","Data":"38b7436d5cc8e10a614405ee498f22240bf4115b092e4c94dca5004fb5b71f31"}
Oct 30 22:09:41.851635 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.851604    4473 generic.go:356] "Generic (PLEG): container finished" podID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerID="ff901a7099a1ec48733fb834cdb168dcac5d9a738c2d88547b3e72e1f17eca85" exitCode=0
Oct 30 22:09:41.851745 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.851668    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" event={"ID":"fa3e75b0-ddd8-481d-9bca-2e10f0b28061","Type":"ContainerDied","Data":"ff901a7099a1ec48733fb834cdb168dcac5d9a738c2d88547b3e72e1f17eca85"}
Oct 30 22:09:41.851745 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.851708    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"
Oct 30 22:09:41.851745 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.851718    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk" event={"ID":"fa3e75b0-ddd8-481d-9bca-2e10f0b28061","Type":"ContainerDied","Data":"f49f3fc609581a265e46cbf0120ad3509ef10cdb319cc59b98d5fc0af49a56be"}
Oct 30 22:09:41.851859 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.851748    4473 scope.go:122] "RemoveContainer" containerID="ff901a7099a1ec48733fb834cdb168dcac5d9a738c2d88547b3e72e1f17eca85"
Oct 30 22:09:41.861659 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.861641    4473 scope.go:122] "RemoveContainer" containerID="959e027195ba0d43ecda0574d81a2fa74bd97d873ef50f84beebccf0bbf2c480"
Oct 30 22:09:41.869506 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.869485    4473 scope.go:122] "RemoveContainer" containerID="e329ed3988a516737d9d3749c5426c6207435998444628c38276e5fe8338289b"
Oct 30 22:09:41.878937 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.878915    4473 scope.go:122] "RemoveContainer" containerID="fba8661085b93b093ea7cc1184b864a041d8d9c5366ee751e3c5acf0ab74e2ea"
Oct 30 22:09:41.886914 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.886892    4473 scope.go:122] "RemoveContainer" containerID="ff901a7099a1ec48733fb834cdb168dcac5d9a738c2d88547b3e72e1f17eca85"
Oct 30 22:09:41.887203 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:09:41.887184    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff901a7099a1ec48733fb834cdb168dcac5d9a738c2d88547b3e72e1f17eca85\": container with ID starting with ff901a7099a1ec48733fb834cdb168dcac5d9a738c2d88547b3e72e1f17eca85 not found: ID does not exist" containerID="ff901a7099a1ec48733fb834cdb168dcac5d9a738c2d88547b3e72e1f17eca85"
Oct 30 22:09:41.887269 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.887210    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff901a7099a1ec48733fb834cdb168dcac5d9a738c2d88547b3e72e1f17eca85"} err="failed to get container status \"ff901a7099a1ec48733fb834cdb168dcac5d9a738c2d88547b3e72e1f17eca85\": rpc error: code = NotFound desc = could not find container \"ff901a7099a1ec48733fb834cdb168dcac5d9a738c2d88547b3e72e1f17eca85\": container with ID starting with ff901a7099a1ec48733fb834cdb168dcac5d9a738c2d88547b3e72e1f17eca85 not found: ID does not exist"
Oct 30 22:09:41.887269 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.887227    4473 scope.go:122] "RemoveContainer" containerID="959e027195ba0d43ecda0574d81a2fa74bd97d873ef50f84beebccf0bbf2c480"
Oct 30 22:09:41.887474 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:09:41.887453    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959e027195ba0d43ecda0574d81a2fa74bd97d873ef50f84beebccf0bbf2c480\": container with ID starting with 959e027195ba0d43ecda0574d81a2fa74bd97d873ef50f84beebccf0bbf2c480 not found: ID does not exist" containerID="959e027195ba0d43ecda0574d81a2fa74bd97d873ef50f84beebccf0bbf2c480"
Oct 30 22:09:41.887515 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.887482    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959e027195ba0d43ecda0574d81a2fa74bd97d873ef50f84beebccf0bbf2c480"} err="failed to get container status \"959e027195ba0d43ecda0574d81a2fa74bd97d873ef50f84beebccf0bbf2c480\": rpc error: code = NotFound desc = could not find container \"959e027195ba0d43ecda0574d81a2fa74bd97d873ef50f84beebccf0bbf2c480\": container with ID starting with 959e027195ba0d43ecda0574d81a2fa74bd97d873ef50f84beebccf0bbf2c480 not found: ID does not exist"
Oct 30 22:09:41.887515 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.887506    4473 scope.go:122] "RemoveContainer" containerID="e329ed3988a516737d9d3749c5426c6207435998444628c38276e5fe8338289b"
Oct 30 22:09:41.887754 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:09:41.887737    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e329ed3988a516737d9d3749c5426c6207435998444628c38276e5fe8338289b\": container with ID starting with e329ed3988a516737d9d3749c5426c6207435998444628c38276e5fe8338289b not found: ID does not exist" containerID="e329ed3988a516737d9d3749c5426c6207435998444628c38276e5fe8338289b"
Oct 30 22:09:41.887864 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.887758    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e329ed3988a516737d9d3749c5426c6207435998444628c38276e5fe8338289b"} err="failed to get container status \"e329ed3988a516737d9d3749c5426c6207435998444628c38276e5fe8338289b\": rpc error: code = NotFound desc = could not find container \"e329ed3988a516737d9d3749c5426c6207435998444628c38276e5fe8338289b\": container with ID starting with e329ed3988a516737d9d3749c5426c6207435998444628c38276e5fe8338289b not found: ID does not exist"
Oct 30 22:09:41.887864 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.887782    4473 scope.go:122] "RemoveContainer" containerID="fba8661085b93b093ea7cc1184b864a041d8d9c5366ee751e3c5acf0ab74e2ea"
Oct 30 22:09:41.888091 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:09:41.888075    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba8661085b93b093ea7cc1184b864a041d8d9c5366ee751e3c5acf0ab74e2ea\": container with ID starting with fba8661085b93b093ea7cc1184b864a041d8d9c5366ee751e3c5acf0ab74e2ea not found: ID does not exist" containerID="fba8661085b93b093ea7cc1184b864a041d8d9c5366ee751e3c5acf0ab74e2ea"
Oct 30 22:09:41.888141 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:41.888096    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba8661085b93b093ea7cc1184b864a041d8d9c5366ee751e3c5acf0ab74e2ea"} err="failed to get container status \"fba8661085b93b093ea7cc1184b864a041d8d9c5366ee751e3c5acf0ab74e2ea\": rpc error: code = NotFound desc = could not find container \"fba8661085b93b093ea7cc1184b864a041d8d9c5366ee751e3c5acf0ab74e2ea\": container with ID starting with fba8661085b93b093ea7cc1184b864a041d8d9c5366ee751e3c5acf0ab74e2ea not found: ID does not exist"
Oct 30 22:09:42.084117 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:42.084080    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"]
Oct 30 22:09:42.087307 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:42.087280    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-sklearn-batcher-custom-predictor-85cf5c46d-8qnlk"]
Oct 30 22:09:43.016631 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:43.016556    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:43.306964 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:43.306880    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:43.889539 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:43.889501    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" path="/var/lib/kubelet/pods/fa3e75b0-ddd8-481d-9bca-2e10f0b28061/volumes"
Oct 30 22:09:49.883156 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:49.883120    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:50.111922 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:50.111887    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:50.266667 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:50.266631    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:53.209205 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:53.209168    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:53.362730 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:53.362699    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:54.223801 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:54.223755    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:56.952245 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:09:56.952208    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:08.995007 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:08.994972    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" event={"ID":"1d96451a-9e25-46de-8dc9-7dc91a5a8120","Type":"ContainerStarted","Data":"a70c7374d722e576210cfd11c2cf313f7697e5a32d31817d86b913d8d4f85a67"}
Oct 30 22:10:08.995427 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:08.995033    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" event={"ID":"1d96451a-9e25-46de-8dc9-7dc91a5a8120","Type":"ContainerStarted","Data":"23b893434668006e43f9c0a082db11c44b879e21743a94786c4c22bb5ec70227"}
Oct 30 22:10:08.995427 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:08.995232    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:10:09.999412 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:09.999382    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:10:10.000749 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:10.000723    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 22:10:11.003289 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:11.003255    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 22:10:11.303265 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:10:11.303187    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:10:16.008433 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:16.008361    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:10:16.009150 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:16.009123    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 22:10:16.179341 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:16.179288    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" podStartSLOduration=9.905401832999999 podStartE2EDuration="40.179274219s" totalImagesPullingTime="26.392728707s" totalInitContainerRuntime="3.881143679s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:09.201781084 +0000 UTC m=+1027.437990429" watchObservedRunningTime="2026-10-30 22:10:16.179274219 +0000 UTC m=+1034.415483555"
Oct 30 22:10:20.417245 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.417207    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-wh56p-sd89g"]
Oct 30 22:10:20.417851 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.417681    4473 cpu_manager.go:414] "containerMap: removing container" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="storage-initializer"
Oct 30 22:10:20.417851 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.417698    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="storage-initializer"
Oct 30 22:10:20.417851 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.417715    4473 cpu_manager.go:414] "containerMap: removing container" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="agent"
Oct 30 22:10:20.417851 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.417723    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="agent"
Oct 30 22:10:20.417851 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.417751    4473 cpu_manager.go:414] "containerMap: removing container" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kube-rbac-proxy"
Oct 30 22:10:20.417851 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.417758    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kube-rbac-proxy"
Oct 30 22:10:20.417851 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.417780    4473 cpu_manager.go:414] "containerMap: removing container" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kserve-container"
Oct 30 22:10:20.417851 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.417788    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kserve-container"
Oct 30 22:10:20.418284 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.417868    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="agent"
Oct 30 22:10:20.418284 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.417882    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kube-rbac-proxy"
Oct 30 22:10:20.418284 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.417893    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa3e75b0-ddd8-481d-9bca-2e10f0b28061" containerName="kserve-container"
Oct 30 22:10:20.431035 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.431005    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wh56p-sd89g"
Oct 30 22:10:20.478800 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.478771    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wh56p-sd89g"]
Oct 30 22:10:20.511117 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.511089    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4a781792-3e14-4e03-b9ef-e29f2c229cd5-archives-path\") pod \"periodic-gathering-wh56p-sd89g\" (UID: \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\") " pod="openshift-insights/periodic-gathering-wh56p-sd89g"
Oct 30 22:10:20.511271 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.511137    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a781792-3e14-4e03-b9ef-e29f2c229cd5-service-ca-bundle\") pod \"periodic-gathering-wh56p-sd89g\" (UID: \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\") " pod="openshift-insights/periodic-gathering-wh56p-sd89g"
Oct 30 22:10:20.511271 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.511161    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlc92\" (UniqueName: \"kubernetes.io/projected/4a781792-3e14-4e03-b9ef-e29f2c229cd5-kube-api-access-tlc92\") pod \"periodic-gathering-wh56p-sd89g\" (UID: \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\") " pod="openshift-insights/periodic-gathering-wh56p-sd89g"
Oct 30 22:10:20.511352 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.511282    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a781792-3e14-4e03-b9ef-e29f2c229cd5-trusted-ca-bundle\") pod \"periodic-gathering-wh56p-sd89g\" (UID: \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\") " pod="openshift-insights/periodic-gathering-wh56p-sd89g"
Oct 30 22:10:20.527674 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.527653    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 22:10:20.527776 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.527702    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jrvhp\""
Oct 30 22:10:20.611894 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.611864    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a781792-3e14-4e03-b9ef-e29f2c229cd5-trusted-ca-bundle\") pod \"periodic-gathering-wh56p-sd89g\" (UID: \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\") " pod="openshift-insights/periodic-gathering-wh56p-sd89g"
Oct 30 22:10:20.612078 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.611920    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4a781792-3e14-4e03-b9ef-e29f2c229cd5-archives-path\") pod \"periodic-gathering-wh56p-sd89g\" (UID: \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\") " pod="openshift-insights/periodic-gathering-wh56p-sd89g"
Oct 30 22:10:20.612078 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.611965    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a781792-3e14-4e03-b9ef-e29f2c229cd5-service-ca-bundle\") pod \"periodic-gathering-wh56p-sd89g\" (UID: \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\") " pod="openshift-insights/periodic-gathering-wh56p-sd89g"
Oct 30 22:10:20.612078 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.612034    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlc92\" (UniqueName: \"kubernetes.io/projected/4a781792-3e14-4e03-b9ef-e29f2c229cd5-kube-api-access-tlc92\") pod \"periodic-gathering-wh56p-sd89g\" (UID: \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\") " pod="openshift-insights/periodic-gathering-wh56p-sd89g"
Oct 30 22:10:20.612387 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.612368    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4a781792-3e14-4e03-b9ef-e29f2c229cd5-archives-path\") pod \"periodic-gathering-wh56p-sd89g\" (UID: \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\") " pod="openshift-insights/periodic-gathering-wh56p-sd89g"
Oct 30 22:10:20.612595 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.612579    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a781792-3e14-4e03-b9ef-e29f2c229cd5-service-ca-bundle\") pod \"periodic-gathering-wh56p-sd89g\" (UID: \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\") " pod="openshift-insights/periodic-gathering-wh56p-sd89g"
Oct 30 22:10:20.686742 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.686663    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 22:10:20.693387 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.693366    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a781792-3e14-4e03-b9ef-e29f2c229cd5-trusted-ca-bundle\") pod \"periodic-gathering-wh56p-sd89g\" (UID: \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\") " pod="openshift-insights/periodic-gathering-wh56p-sd89g"
Oct 30 22:10:20.733036 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.733007    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlc92\" (UniqueName: \"kubernetes.io/projected/4a781792-3e14-4e03-b9ef-e29f2c229cd5-kube-api-access-tlc92\") pod \"periodic-gathering-wh56p-sd89g\" (UID: \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\") " pod="openshift-insights/periodic-gathering-wh56p-sd89g"
Oct 30 22:10:20.753879 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:20.753848    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wh56p-sd89g"
Oct 30 22:10:21.154123 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:21.154097    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wh56p-sd89g"]
Oct 30 22:10:21.156649 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:10:21.156622    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a781792_3e14_4e03_b9ef_e29f2c229cd5.slice/crio-259a25bd2710c1b02ff64b91bcf42534824f92710fa7d0a40d9963b88b835317 WatchSource:0}: Error finding container 259a25bd2710c1b02ff64b91bcf42534824f92710fa7d0a40d9963b88b835317: Status 404 returned error can't find the container with id 259a25bd2710c1b02ff64b91bcf42534824f92710fa7d0a40d9963b88b835317
Oct 30 22:10:22.047827 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:22.047791    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wh56p-sd89g" event={"ID":"4a781792-3e14-4e03-b9ef-e29f2c229cd5","Type":"ContainerStarted","Data":"621daad1d52e090edef8a7566e32e9b70374c867405ae04864124159925eb5cb"}
Oct 30 22:10:22.048244 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:22.047848    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wh56p-sd89g" event={"ID":"4a781792-3e14-4e03-b9ef-e29f2c229cd5","Type":"ContainerStarted","Data":"259a25bd2710c1b02ff64b91bcf42534824f92710fa7d0a40d9963b88b835317"}
Oct 30 22:10:22.164266 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:22.164207    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-wh56p-sd89g" podStartSLOduration=2.164192478 podStartE2EDuration="2.164192478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:22.163318798 +0000 UTC m=+1040.399528143" watchObservedRunningTime="2026-10-30 22:10:22.164192478 +0000 UTC m=+1040.400401821"
Oct 30 22:10:26.010000 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:26.009943    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 22:10:36.009808 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:36.009764    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 22:10:41.303088 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:10:41.303053    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:10:46.009575 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:46.009530    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 22:10:56.009784 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:56.009746    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 22:10:57.299106 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:10:57.299076    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:00.610100 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:00.610065    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:00.783571 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:00.783527    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:01.766309 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:01.766263    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:06.009879 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:06.009834    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 22:11:11.303422 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:11:11.303388    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:11:16.010188 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:16.010155    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:11:27.529263 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:27.529228    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"]
Oct 30 22:11:27.529664 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:27.529491    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="kserve-container" containerID="cri-o://23b893434668006e43f9c0a082db11c44b879e21743a94786c4c22bb5ec70227" gracePeriod=30
Oct 30 22:11:27.529664 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:27.529556    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="kube-rbac-proxy" containerID="cri-o://a70c7374d722e576210cfd11c2cf313f7697e5a32d31817d86b913d8d4f85a67" gracePeriod=30
Oct 30 22:11:27.549189 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:27.549163    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:28.310029 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:28.309994    4473 generic.go:356] "Generic (PLEG): container finished" podID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerID="a70c7374d722e576210cfd11c2cf313f7697e5a32d31817d86b913d8d4f85a67" exitCode=0
Oct 30 22:11:28.310197 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:28.310068    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" event={"ID":"1d96451a-9e25-46de-8dc9-7dc91a5a8120","Type":"ContainerDied","Data":"a70c7374d722e576210cfd11c2cf313f7697e5a32d31817d86b913d8d4f85a67"}
Oct 30 22:11:31.003877 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:31.003833    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Oct 30 22:11:32.270508 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.270486    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:11:32.328363 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.328285    4473 generic.go:356] "Generic (PLEG): container finished" podID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerID="23b893434668006e43f9c0a082db11c44b879e21743a94786c4c22bb5ec70227" exitCode=0
Oct 30 22:11:32.328363 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.328346    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"
Oct 30 22:11:32.328524 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.328348    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" event={"ID":"1d96451a-9e25-46de-8dc9-7dc91a5a8120","Type":"ContainerDied","Data":"23b893434668006e43f9c0a082db11c44b879e21743a94786c4c22bb5ec70227"}
Oct 30 22:11:32.328524 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.328480    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8" event={"ID":"1d96451a-9e25-46de-8dc9-7dc91a5a8120","Type":"ContainerDied","Data":"d6ef99cb1c09cec279e16f32d184167181598c467315f93c9ca1b7b2bc0ff455"}
Oct 30 22:11:32.328524 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.328505    4473 scope.go:122] "RemoveContainer" containerID="a70c7374d722e576210cfd11c2cf313f7697e5a32d31817d86b913d8d4f85a67"
Oct 30 22:11:32.336280 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.336264    4473 scope.go:122] "RemoveContainer" containerID="23b893434668006e43f9c0a082db11c44b879e21743a94786c4c22bb5ec70227"
Oct 30 22:11:32.343338 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.343322    4473 scope.go:122] "RemoveContainer" containerID="38b7436d5cc8e10a614405ee498f22240bf4115b092e4c94dca5004fb5b71f31"
Oct 30 22:11:32.350967 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.350922    4473 scope.go:122] "RemoveContainer" containerID="a70c7374d722e576210cfd11c2cf313f7697e5a32d31817d86b913d8d4f85a67"
Oct 30 22:11:32.351244 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:11:32.351228    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70c7374d722e576210cfd11c2cf313f7697e5a32d31817d86b913d8d4f85a67\": container with ID starting with a70c7374d722e576210cfd11c2cf313f7697e5a32d31817d86b913d8d4f85a67 not found: ID does not exist" containerID="a70c7374d722e576210cfd11c2cf313f7697e5a32d31817d86b913d8d4f85a67"
Oct 30 22:11:32.351290 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.351254    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70c7374d722e576210cfd11c2cf313f7697e5a32d31817d86b913d8d4f85a67"} err="failed to get container status \"a70c7374d722e576210cfd11c2cf313f7697e5a32d31817d86b913d8d4f85a67\": rpc error: code = NotFound desc = could not find container \"a70c7374d722e576210cfd11c2cf313f7697e5a32d31817d86b913d8d4f85a67\": container with ID starting with a70c7374d722e576210cfd11c2cf313f7697e5a32d31817d86b913d8d4f85a67 not found: ID does not exist"
Oct 30 22:11:32.351290 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.351271    4473 scope.go:122] "RemoveContainer" containerID="23b893434668006e43f9c0a082db11c44b879e21743a94786c4c22bb5ec70227"
Oct 30 22:11:32.351545 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:11:32.351522    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b893434668006e43f9c0a082db11c44b879e21743a94786c4c22bb5ec70227\": container with ID starting with 23b893434668006e43f9c0a082db11c44b879e21743a94786c4c22bb5ec70227 not found: ID does not exist" containerID="23b893434668006e43f9c0a082db11c44b879e21743a94786c4c22bb5ec70227"
Oct 30 22:11:32.351646 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.351549    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b893434668006e43f9c0a082db11c44b879e21743a94786c4c22bb5ec70227"} err="failed to get container status \"23b893434668006e43f9c0a082db11c44b879e21743a94786c4c22bb5ec70227\": rpc error: code = NotFound desc = could not find container \"23b893434668006e43f9c0a082db11c44b879e21743a94786c4c22bb5ec70227\": container with ID starting with 23b893434668006e43f9c0a082db11c44b879e21743a94786c4c22bb5ec70227 not found: ID does not exist"
Oct 30 22:11:32.351646 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.351564    4473 scope.go:122] "RemoveContainer" containerID="38b7436d5cc8e10a614405ee498f22240bf4115b092e4c94dca5004fb5b71f31"
Oct 30 22:11:32.351793 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:11:32.351773    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b7436d5cc8e10a614405ee498f22240bf4115b092e4c94dca5004fb5b71f31\": container with ID starting with 38b7436d5cc8e10a614405ee498f22240bf4115b092e4c94dca5004fb5b71f31 not found: ID does not exist" containerID="38b7436d5cc8e10a614405ee498f22240bf4115b092e4c94dca5004fb5b71f31"
Oct 30 22:11:32.351834 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.351798    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b7436d5cc8e10a614405ee498f22240bf4115b092e4c94dca5004fb5b71f31"} err="failed to get container status \"38b7436d5cc8e10a614405ee498f22240bf4115b092e4c94dca5004fb5b71f31\": rpc error: code = NotFound desc = could not find container \"38b7436d5cc8e10a614405ee498f22240bf4115b092e4c94dca5004fb5b71f31\": container with ID starting with 38b7436d5cc8e10a614405ee498f22240bf4115b092e4c94dca5004fb5b71f31 not found: ID does not exist"
Oct 30 22:11:32.378868 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.378840    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kube-api-access-ndnsf\" (UniqueName: \"kubernetes.io/projected/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kube-api-access-ndnsf\") pod \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") "
Oct 30 22:11:32.379029 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.378887    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kserve-provision-location\") pod \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") "
Oct 30 22:11:32.379112 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.379029    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d96451a-9e25-46de-8dc9-7dc91a5a8120-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d96451a-9e25-46de-8dc9-7dc91a5a8120-proxy-tls\") pod \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") "
Oct 30 22:11:32.379232 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.379128    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d96451a-9e25-46de-8dc9-7dc91a5a8120-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d96451a-9e25-46de-8dc9-7dc91a5a8120-cabundle-cert\") pod \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") "
Oct 30 22:11:32.379232 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.379165    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kube-rbac-proxy-sar-config\") pod \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\" (UID: \"1d96451a-9e25-46de-8dc9-7dc91a5a8120\") "
Oct 30 22:11:32.379334 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.379237    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kserve-provision-location" pod "1d96451a-9e25-46de-8dc9-7dc91a5a8120" (UID: "1d96451a-9e25-46de-8dc9-7dc91a5a8120"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:32.379454 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.379437    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:32.379541 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.379518    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d96451a-9e25-46de-8dc9-7dc91a5a8120-cabundle-cert" pod "1d96451a-9e25-46de-8dc9-7dc91a5a8120" (UID: "1d96451a-9e25-46de-8dc9-7dc91a5a8120"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:32.379580 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.379527    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kube-rbac-proxy-sar-config" pod "1d96451a-9e25-46de-8dc9-7dc91a5a8120" (UID: "1d96451a-9e25-46de-8dc9-7dc91a5a8120"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:32.381187 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.381169    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kube-api-access-ndnsf" pod "1d96451a-9e25-46de-8dc9-7dc91a5a8120" (UID: "1d96451a-9e25-46de-8dc9-7dc91a5a8120"). InnerVolumeSpecName "kube-api-access-ndnsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:32.381244 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.381181    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d96451a-9e25-46de-8dc9-7dc91a5a8120-proxy-tls" pod "1d96451a-9e25-46de-8dc9-7dc91a5a8120" (UID: "1d96451a-9e25-46de-8dc9-7dc91a5a8120"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:32.480666 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.480636    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:32.480666 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.480664    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndnsf\" (UniqueName: \"kubernetes.io/projected/1d96451a-9e25-46de-8dc9-7dc91a5a8120-kube-api-access-ndnsf\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:32.480864 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.480696    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d96451a-9e25-46de-8dc9-7dc91a5a8120-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:32.480864 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.480706    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d96451a-9e25-46de-8dc9-7dc91a5a8120-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:32.753359 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.753312    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"]
Oct 30 22:11:32.756792 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:32.756767    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-sklearn-predictor-7657bc4bdf-5wcz8"]
Oct 30 22:11:33.698535 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:33.698502    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:33.855494 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:33.855463    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:33.886427 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:33.886396    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" path="/var/lib/kubelet/pods/1d96451a-9e25-46de-8dc9-7dc91a5a8120/volumes"
Oct 30 22:11:34.608691 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:34.608659    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:34.835593 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:34.835562    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:36.117416 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.117383    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:36.333005 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.332947    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:36.647436 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.647402    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"]
Oct 30 22:11:36.647811 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.647784    4473 cpu_manager.go:414] "containerMap: removing container" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="kube-rbac-proxy"
Oct 30 22:11:36.647811 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.647807    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="kube-rbac-proxy"
Oct 30 22:11:36.648008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.647822    4473 cpu_manager.go:414] "containerMap: removing container" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="storage-initializer"
Oct 30 22:11:36.648008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.647832    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="storage-initializer"
Oct 30 22:11:36.648008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.647849    4473 cpu_manager.go:414] "containerMap: removing container" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="kserve-container"
Oct 30 22:11:36.648008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.647854    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="kserve-container"
Oct 30 22:11:36.648008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.647907    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="kserve-container"
Oct 30 22:11:36.648008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.647919    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d96451a-9e25-46de-8dc9-7dc91a5a8120" containerName="kube-rbac-proxy"
Oct 30 22:11:36.650506 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.650492    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:36.704040 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.704013    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:11:36.718362 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.718333    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72f1380e-1dde-476d-8569-375f0bab5235-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:36.718495 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.718379    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72f1380e-1dde-476d-8569-375f0bab5235-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:36.718495 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.718405    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72f1380e-1dde-476d-8569-375f0bab5235-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:36.718495 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.718479    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27mf4\" (UniqueName: \"kubernetes.io/projected/72f1380e-1dde-476d-8569-375f0bab5235-kube-api-access-27mf4\") pod \"isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:36.718608 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.718518    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72f1380e-1dde-476d-8569-375f0bab5235-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:36.754234 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.754190    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Oct 30 22:11:36.754234 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.754190    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:11:36.754430 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.754283    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:11:36.754430 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.754328    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:11:36.754430 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.754409    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Oct 30 22:11:36.754522 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.754433    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:11:36.756052 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.756036    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"]
Oct 30 22:11:36.819315 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.819283    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72f1380e-1dde-476d-8569-375f0bab5235-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:36.819315 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.819318    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72f1380e-1dde-476d-8569-375f0bab5235-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:36.819573 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.819339    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27mf4\" (UniqueName: \"kubernetes.io/projected/72f1380e-1dde-476d-8569-375f0bab5235-kube-api-access-27mf4\") pod \"isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:36.819573 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.819370    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72f1380e-1dde-476d-8569-375f0bab5235-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:36.819573 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.819415    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72f1380e-1dde-476d-8569-375f0bab5235-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:36.819879 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.819853    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72f1380e-1dde-476d-8569-375f0bab5235-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:36.820135 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.820115    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72f1380e-1dde-476d-8569-375f0bab5235-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:36.820198 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.820121    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72f1380e-1dde-476d-8569-375f0bab5235-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:36.822028 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.822007    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72f1380e-1dde-476d-8569-375f0bab5235-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:36.877478 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.877446    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27mf4\" (UniqueName: \"kubernetes.io/projected/72f1380e-1dde-476d-8569-375f0bab5235-kube-api-access-27mf4\") pod \"isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:36.963525 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:36.963444    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:37.385548 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:37.385521    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"]
Oct 30 22:11:37.388308 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:11:37.388281    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f1380e_1dde_476d_8569_375f0bab5235.slice/crio-1d72af197dd916bb9235d0ab6ecfa3772124c45216968eb3ca94cc1fa1a7b617 WatchSource:0}: Error finding container 1d72af197dd916bb9235d0ab6ecfa3772124c45216968eb3ca94cc1fa1a7b617: Status 404 returned error can't find the container with id 1d72af197dd916bb9235d0ab6ecfa3772124c45216968eb3ca94cc1fa1a7b617
Oct 30 22:11:38.357181 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:38.357146    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" event={"ID":"72f1380e-1dde-476d-8569-375f0bab5235","Type":"ContainerStarted","Data":"4a944a298e7e68c156f71728400c79fb2bfc71e3240468d6d21704eb235a1e33"}
Oct 30 22:11:38.357360 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:38.357249    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" event={"ID":"72f1380e-1dde-476d-8569-375f0bab5235","Type":"ContainerStarted","Data":"1d72af197dd916bb9235d0ab6ecfa3772124c45216968eb3ca94cc1fa1a7b617"}
Oct 30 22:11:39.166623 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:39.166576    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:39.390296 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:39.390259    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:39.575579 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:39.575487    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:41.303782 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:11:41.303741    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:11:41.372480 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:41.372448    4473 generic.go:356] "Generic (PLEG): container finished" podID="72f1380e-1dde-476d-8569-375f0bab5235" containerID="4a944a298e7e68c156f71728400c79fb2bfc71e3240468d6d21704eb235a1e33" exitCode=0
Oct 30 22:11:41.372659 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:41.372517    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" event={"ID":"72f1380e-1dde-476d-8569-375f0bab5235","Type":"ContainerDied","Data":"4a944a298e7e68c156f71728400c79fb2bfc71e3240468d6d21704eb235a1e33"}
Oct 30 22:11:41.679810 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:41.679770    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:41.841400 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:41.841369    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:42.378554 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:42.378480    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" event={"ID":"72f1380e-1dde-476d-8569-375f0bab5235","Type":"ContainerStarted","Data":"bdc8b72013dd0f6b48d843c9868d877f82419ff6e4222031c7f50e5d36801e03"}
Oct 30 22:11:42.378554 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:42.378538    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" event={"ID":"72f1380e-1dde-476d-8569-375f0bab5235","Type":"ContainerStarted","Data":"50ec0c22618c926f5ae494b45afdfe7c581f88e584143db4c6ccaad23b6afaf9"}
Oct 30 22:11:42.379000 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:42.378812    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:42.379000 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:42.378944    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:42.380297 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:42.380271    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 30 22:11:42.551735 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:42.551687    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" podStartSLOduration=2.868240701 podStartE2EDuration="6.551673673s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.683432972s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:42.550550123 +0000 UTC m=+1120.786759459" watchObservedRunningTime="2026-10-30 22:11:42.551673673 +0000 UTC m=+1120.787883018"
Oct 30 22:11:42.598818 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:42.598786    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:43.382361 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:43.382324    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 30 22:11:48.387568 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:48.387539    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:11:48.388108 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:48.388083    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 30 22:11:48.929513 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:48.929469    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:49.109014 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:49.108971    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:51.415276 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:51.415242    4473 generic.go:356] "Generic (PLEG): container finished" podID="4a781792-3e14-4e03-b9ef-e29f2c229cd5" containerID="621daad1d52e090edef8a7566e32e9b70374c867405ae04864124159925eb5cb" exitCode=0
Oct 30 22:11:51.415656 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:51.415293    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wh56p-sd89g" event={"ID":"4a781792-3e14-4e03-b9ef-e29f2c229cd5","Type":"ContainerDied","Data":"621daad1d52e090edef8a7566e32e9b70374c867405ae04864124159925eb5cb"}
Oct 30 22:11:52.554078 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:52.554054    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wh56p-sd89g"
Oct 30 22:11:52.647711 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:52.647669    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a781792-3e14-4e03-b9ef-e29f2c229cd5-kube-api-access-tlc92\" (UniqueName: \"kubernetes.io/projected/4a781792-3e14-4e03-b9ef-e29f2c229cd5-kube-api-access-tlc92\") pod \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\" (UID: \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\") "
Oct 30 22:11:52.647898 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:52.647730    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a781792-3e14-4e03-b9ef-e29f2c229cd5-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a781792-3e14-4e03-b9ef-e29f2c229cd5-service-ca-bundle\") pod \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\" (UID: \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\") "
Oct 30 22:11:52.647898 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:52.647845    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a781792-3e14-4e03-b9ef-e29f2c229cd5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a781792-3e14-4e03-b9ef-e29f2c229cd5-trusted-ca-bundle\") pod \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\" (UID: \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\") "
Oct 30 22:11:52.648022 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:52.647895    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a781792-3e14-4e03-b9ef-e29f2c229cd5-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4a781792-3e14-4e03-b9ef-e29f2c229cd5-archives-path\") pod \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\" (UID: \"4a781792-3e14-4e03-b9ef-e29f2c229cd5\") "
Oct 30 22:11:52.648228 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:52.648195    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a781792-3e14-4e03-b9ef-e29f2c229cd5-service-ca-bundle" pod "4a781792-3e14-4e03-b9ef-e29f2c229cd5" (UID: "4a781792-3e14-4e03-b9ef-e29f2c229cd5"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:52.648383 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:52.648258    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a781792-3e14-4e03-b9ef-e29f2c229cd5-archives-path" pod "4a781792-3e14-4e03-b9ef-e29f2c229cd5" (UID: "4a781792-3e14-4e03-b9ef-e29f2c229cd5"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:52.648383 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:52.648256    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a781792-3e14-4e03-b9ef-e29f2c229cd5-trusted-ca-bundle" pod "4a781792-3e14-4e03-b9ef-e29f2c229cd5" (UID: "4a781792-3e14-4e03-b9ef-e29f2c229cd5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:52.650154 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:52.650134    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a781792-3e14-4e03-b9ef-e29f2c229cd5-kube-api-access-tlc92" pod "4a781792-3e14-4e03-b9ef-e29f2c229cd5" (UID: "4a781792-3e14-4e03-b9ef-e29f2c229cd5"). InnerVolumeSpecName "kube-api-access-tlc92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:52.748984 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:52.748882    4473 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a781792-3e14-4e03-b9ef-e29f2c229cd5-trusted-ca-bundle\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:52.748984 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:52.748908    4473 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4a781792-3e14-4e03-b9ef-e29f2c229cd5-archives-path\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:52.748984 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:52.748918    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlc92\" (UniqueName: \"kubernetes.io/projected/4a781792-3e14-4e03-b9ef-e29f2c229cd5-kube-api-access-tlc92\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:52.748984 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:52.748929    4473 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a781792-3e14-4e03-b9ef-e29f2c229cd5-service-ca-bundle\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:53.424219 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:53.424188    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wh56p-sd89g"
Oct 30 22:11:53.424219 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:53.424188    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wh56p-sd89g" event={"ID":"4a781792-3e14-4e03-b9ef-e29f2c229cd5","Type":"ContainerDied","Data":"259a25bd2710c1b02ff64b91bcf42534824f92710fa7d0a40d9963b88b835317"}
Oct 30 22:11:53.424450 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:53.424249    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="259a25bd2710c1b02ff64b91bcf42534824f92710fa7d0a40d9963b88b835317"
Oct 30 22:11:57.643128 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:57.643085    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:58.388693 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:11:58.388660    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 30 22:12:08.388383 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:08.388343    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 30 22:12:11.303741 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:12:11.303709    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:12:18.388056 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:18.388020    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 30 22:12:28.388593 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:28.388546    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 30 22:12:34.092228 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:34.092187    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:38.381684 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:38.381644    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:38.388104 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:38.388073    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 30 22:12:41.303820 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:12:41.303775    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:12:48.388920 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:48.388891    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:12:56.048070 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:56.048036    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:56.210970 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:56.210918    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:57.389120 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:57.389089    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:57.904461 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:57.904432    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"]
Oct 30 22:12:57.904674 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:57.904649    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kserve-container" containerID="cri-o://50ec0c22618c926f5ae494b45afdfe7c581f88e584143db4c6ccaad23b6afaf9" gracePeriod=30
Oct 30 22:12:57.904765 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:57.904689    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kube-rbac-proxy" containerID="cri-o://bdc8b72013dd0f6b48d843c9868d877f82419ff6e4222031c7f50e5d36801e03" gracePeriod=30
Oct 30 22:12:58.202139 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:58.202057    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:58.383293 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:58.383255    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Oct 30 22:12:58.388587 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:58.388565    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 30 22:12:58.692061 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:58.692031    4473 generic.go:356] "Generic (PLEG): container finished" podID="72f1380e-1dde-476d-8569-375f0bab5235" containerID="bdc8b72013dd0f6b48d843c9868d877f82419ff6e4222031c7f50e5d36801e03" exitCode=0
Oct 30 22:12:58.692446 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:12:58.692078    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" event={"ID":"72f1380e-1dde-476d-8569-375f0bab5235","Type":"ContainerDied","Data":"bdc8b72013dd0f6b48d843c9868d877f82419ff6e4222031c7f50e5d36801e03"}
Oct 30 22:13:00.482019 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:00.481985    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:00.641070 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:00.641032    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:01.985025 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:01.984977    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:13:01.987517 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:01.987491    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:13:02.643974 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.643923    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:13:02.710324 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.710296    4473 generic.go:356] "Generic (PLEG): container finished" podID="72f1380e-1dde-476d-8569-375f0bab5235" containerID="50ec0c22618c926f5ae494b45afdfe7c581f88e584143db4c6ccaad23b6afaf9" exitCode=0
Oct 30 22:13:02.710494 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.710332    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" event={"ID":"72f1380e-1dde-476d-8569-375f0bab5235","Type":"ContainerDied","Data":"50ec0c22618c926f5ae494b45afdfe7c581f88e584143db4c6ccaad23b6afaf9"}
Oct 30 22:13:02.710494 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.710351    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"
Oct 30 22:13:02.710494 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.710396    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn" event={"ID":"72f1380e-1dde-476d-8569-375f0bab5235","Type":"ContainerDied","Data":"1d72af197dd916bb9235d0ab6ecfa3772124c45216968eb3ca94cc1fa1a7b617"}
Oct 30 22:13:02.710494 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.710432    4473 scope.go:122] "RemoveContainer" containerID="bdc8b72013dd0f6b48d843c9868d877f82419ff6e4222031c7f50e5d36801e03"
Oct 30 22:13:02.717690 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.717637    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72f1380e-1dde-476d-8569-375f0bab5235-proxy-tls\" (UniqueName: \"kubernetes.io/secret/72f1380e-1dde-476d-8569-375f0bab5235-proxy-tls\") pod \"72f1380e-1dde-476d-8569-375f0bab5235\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") "
Oct 30 22:13:02.717796 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.717699    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f1380e-1dde-476d-8569-375f0bab5235-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72f1380e-1dde-476d-8569-375f0bab5235-cabundle-cert\") pod \"72f1380e-1dde-476d-8569-375f0bab5235\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") "
Oct 30 22:13:02.717796 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.717750    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f1380e-1dde-476d-8569-375f0bab5235-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72f1380e-1dde-476d-8569-375f0bab5235-kserve-provision-location\") pod \"72f1380e-1dde-476d-8569-375f0bab5235\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") "
Oct 30 22:13:02.717896 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.717799    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f1380e-1dde-476d-8569-375f0bab5235-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72f1380e-1dde-476d-8569-375f0bab5235-kube-rbac-proxy-sar-config\") pod \"72f1380e-1dde-476d-8569-375f0bab5235\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") "
Oct 30 22:13:02.717896 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.717831    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f1380e-1dde-476d-8569-375f0bab5235-kube-api-access-27mf4\" (UniqueName: \"kubernetes.io/projected/72f1380e-1dde-476d-8569-375f0bab5235-kube-api-access-27mf4\") pod \"72f1380e-1dde-476d-8569-375f0bab5235\" (UID: \"72f1380e-1dde-476d-8569-375f0bab5235\") "
Oct 30 22:13:02.718190 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.718157    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f1380e-1dde-476d-8569-375f0bab5235-kserve-provision-location" pod "72f1380e-1dde-476d-8569-375f0bab5235" (UID: "72f1380e-1dde-476d-8569-375f0bab5235"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:02.718190 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.718164    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f1380e-1dde-476d-8569-375f0bab5235-kube-rbac-proxy-sar-config" pod "72f1380e-1dde-476d-8569-375f0bab5235" (UID: "72f1380e-1dde-476d-8569-375f0bab5235"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:02.718495 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.718226    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f1380e-1dde-476d-8569-375f0bab5235-cabundle-cert" pod "72f1380e-1dde-476d-8569-375f0bab5235" (UID: "72f1380e-1dde-476d-8569-375f0bab5235"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:02.718688 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.718667    4473 scope.go:122] "RemoveContainer" containerID="50ec0c22618c926f5ae494b45afdfe7c581f88e584143db4c6ccaad23b6afaf9"
Oct 30 22:13:02.719992 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.719940    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f1380e-1dde-476d-8569-375f0bab5235-proxy-tls" pod "72f1380e-1dde-476d-8569-375f0bab5235" (UID: "72f1380e-1dde-476d-8569-375f0bab5235"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:02.721218 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.721200    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f1380e-1dde-476d-8569-375f0bab5235-kube-api-access-27mf4" pod "72f1380e-1dde-476d-8569-375f0bab5235" (UID: "72f1380e-1dde-476d-8569-375f0bab5235"). InnerVolumeSpecName "kube-api-access-27mf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:02.733745 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.733726    4473 scope.go:122] "RemoveContainer" containerID="4a944a298e7e68c156f71728400c79fb2bfc71e3240468d6d21704eb235a1e33"
Oct 30 22:13:02.740838 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.740818    4473 scope.go:122] "RemoveContainer" containerID="bdc8b72013dd0f6b48d843c9868d877f82419ff6e4222031c7f50e5d36801e03"
Oct 30 22:13:02.741106 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:13:02.741086    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc8b72013dd0f6b48d843c9868d877f82419ff6e4222031c7f50e5d36801e03\": container with ID starting with bdc8b72013dd0f6b48d843c9868d877f82419ff6e4222031c7f50e5d36801e03 not found: ID does not exist" containerID="bdc8b72013dd0f6b48d843c9868d877f82419ff6e4222031c7f50e5d36801e03"
Oct 30 22:13:02.741158 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.741120    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc8b72013dd0f6b48d843c9868d877f82419ff6e4222031c7f50e5d36801e03"} err="failed to get container status \"bdc8b72013dd0f6b48d843c9868d877f82419ff6e4222031c7f50e5d36801e03\": rpc error: code = NotFound desc = could not find container \"bdc8b72013dd0f6b48d843c9868d877f82419ff6e4222031c7f50e5d36801e03\": container with ID starting with bdc8b72013dd0f6b48d843c9868d877f82419ff6e4222031c7f50e5d36801e03 not found: ID does not exist"
Oct 30 22:13:02.741158 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.741144    4473 scope.go:122] "RemoveContainer" containerID="50ec0c22618c926f5ae494b45afdfe7c581f88e584143db4c6ccaad23b6afaf9"
Oct 30 22:13:02.741362 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:13:02.741346    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50ec0c22618c926f5ae494b45afdfe7c581f88e584143db4c6ccaad23b6afaf9\": container with ID starting with 50ec0c22618c926f5ae494b45afdfe7c581f88e584143db4c6ccaad23b6afaf9 not found: ID does not exist" containerID="50ec0c22618c926f5ae494b45afdfe7c581f88e584143db4c6ccaad23b6afaf9"
Oct 30 22:13:02.741402 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.741371    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ec0c22618c926f5ae494b45afdfe7c581f88e584143db4c6ccaad23b6afaf9"} err="failed to get container status \"50ec0c22618c926f5ae494b45afdfe7c581f88e584143db4c6ccaad23b6afaf9\": rpc error: code = NotFound desc = could not find container \"50ec0c22618c926f5ae494b45afdfe7c581f88e584143db4c6ccaad23b6afaf9\": container with ID starting with 50ec0c22618c926f5ae494b45afdfe7c581f88e584143db4c6ccaad23b6afaf9 not found: ID does not exist"
Oct 30 22:13:02.741402 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.741383    4473 scope.go:122] "RemoveContainer" containerID="4a944a298e7e68c156f71728400c79fb2bfc71e3240468d6d21704eb235a1e33"
Oct 30 22:13:02.741617 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:13:02.741599    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a944a298e7e68c156f71728400c79fb2bfc71e3240468d6d21704eb235a1e33\": container with ID starting with 4a944a298e7e68c156f71728400c79fb2bfc71e3240468d6d21704eb235a1e33 not found: ID does not exist" containerID="4a944a298e7e68c156f71728400c79fb2bfc71e3240468d6d21704eb235a1e33"
Oct 30 22:13:02.741655 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.741622    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a944a298e7e68c156f71728400c79fb2bfc71e3240468d6d21704eb235a1e33"} err="failed to get container status \"4a944a298e7e68c156f71728400c79fb2bfc71e3240468d6d21704eb235a1e33\": rpc error: code = NotFound desc = could not find container \"4a944a298e7e68c156f71728400c79fb2bfc71e3240468d6d21704eb235a1e33\": container with ID starting with 4a944a298e7e68c156f71728400c79fb2bfc71e3240468d6d21704eb235a1e33 not found: ID does not exist"
Oct 30 22:13:02.818634 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.818604    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72f1380e-1dde-476d-8569-375f0bab5235-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:02.818784 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.818636    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72f1380e-1dde-476d-8569-375f0bab5235-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:02.818784 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.818655    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72f1380e-1dde-476d-8569-375f0bab5235-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:02.818784 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.818669    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72f1380e-1dde-476d-8569-375f0bab5235-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:02.818784 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:02.818683    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27mf4\" (UniqueName: \"kubernetes.io/projected/72f1380e-1dde-476d-8569-375f0bab5235-kube-api-access-27mf4\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:03.131483 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:03.131452    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"]
Oct 30 22:13:03.135233 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:03.135207    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-xgboost-predictor-7cb5987b8b-gw8qn"]
Oct 30 22:13:03.886311 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:03.886267    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f1380e-1dde-476d-8569-375f0bab5235" path="/var/lib/kubelet/pods/72f1380e-1dde-476d-8569-375f0bab5235/volumes"
Oct 30 22:13:05.751110 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:05.751077    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:06.083981 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.083871    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:06.247599 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.247567    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"]
Oct 30 22:13:06.247902 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.247889    4473 cpu_manager.go:414] "containerMap: removing container" podUID="4a781792-3e14-4e03-b9ef-e29f2c229cd5" containerName="insights-gathering"
Oct 30 22:13:06.247963 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.247905    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a781792-3e14-4e03-b9ef-e29f2c229cd5" containerName="insights-gathering"
Oct 30 22:13:06.247963 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.247916    4473 cpu_manager.go:414] "containerMap: removing container" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kube-rbac-proxy"
Oct 30 22:13:06.247963 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.247923    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kube-rbac-proxy"
Oct 30 22:13:06.247963 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.247941    4473 cpu_manager.go:414] "containerMap: removing container" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="storage-initializer"
Oct 30 22:13:06.247963 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.247962    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="storage-initializer"
Oct 30 22:13:06.248114 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.247972    4473 cpu_manager.go:414] "containerMap: removing container" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kserve-container"
Oct 30 22:13:06.248114 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.247977    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kserve-container"
Oct 30 22:13:06.248114 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.248034    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kserve-container"
Oct 30 22:13:06.248114 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.248048    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a781792-3e14-4e03-b9ef-e29f2c229cd5" containerName="insights-gathering"
Oct 30 22:13:06.248114 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.248054    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f1380e-1dde-476d-8569-375f0bab5235" containerName="kube-rbac-proxy"
Oct 30 22:13:06.252765 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.252741    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:06.306038 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.306011    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:13:06.347886 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.347814    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-db87765bc-29jrf\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:06.347886 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.347860    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-db87765bc-29jrf\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:06.347886 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.347884    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-db87765bc-29jrf\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:06.348243 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.348060    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-db87765bc-29jrf\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:06.348243 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.348114    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvlmc\" (UniqueName: \"kubernetes.io/projected/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kube-api-access-pvlmc\") pod \"isvc-predictive-lightgbm-predictor-db87765bc-29jrf\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:06.356553 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.356531    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:13:06.356750 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.356727    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:13:06.356835 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.356765    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Oct 30 22:13:06.356835 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.356777    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:13:06.356835 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.356764    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:13:06.357014 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.356854    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Oct 30 22:13:06.358158 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.358137    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"]
Oct 30 22:13:06.448558 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.448525    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-db87765bc-29jrf\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:06.448732 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.448570    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvlmc\" (UniqueName: \"kubernetes.io/projected/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kube-api-access-pvlmc\") pod \"isvc-predictive-lightgbm-predictor-db87765bc-29jrf\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:06.448732 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.448618    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-db87765bc-29jrf\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:06.448732 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.448646    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-db87765bc-29jrf\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:06.448732 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.448697    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-db87765bc-29jrf\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:06.449072 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.449048    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-db87765bc-29jrf\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:06.449268 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.449249    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-db87765bc-29jrf\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:06.449326 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.449300    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-db87765bc-29jrf\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:06.451472 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.451446    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-db87765bc-29jrf\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:06.506503 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.506474    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvlmc\" (UniqueName: \"kubernetes.io/projected/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kube-api-access-pvlmc\") pod \"isvc-predictive-lightgbm-predictor-db87765bc-29jrf\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:06.569493 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.569458    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:06.992507 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:06.992475    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"]
Oct 30 22:13:07.032599 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:13:07.032573    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0887f84_01cf_4b06_aadf_88fb49a5e2ad.slice/crio-26ddb4b50e7073b6456824407155895e6459dfecbb984796d6639d58c99842ce WatchSource:0}: Error finding container 26ddb4b50e7073b6456824407155895e6459dfecbb984796d6639d58c99842ce: Status 404 returned error can't find the container with id 26ddb4b50e7073b6456824407155895e6459dfecbb984796d6639d58c99842ce
Oct 30 22:13:07.732830 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:07.732789    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf" event={"ID":"a0887f84-01cf-4b06-aadf-88fb49a5e2ad","Type":"ContainerStarted","Data":"e62a25ef884dd287e32ca5b1d6b028f8c9d47e70b0b6d418d3605558296ef5c9"}
Oct 30 22:13:07.733037 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:07.732863    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf" event={"ID":"a0887f84-01cf-4b06-aadf-88fb49a5e2ad","Type":"ContainerStarted","Data":"26ddb4b50e7073b6456824407155895e6459dfecbb984796d6639d58c99842ce"}
Oct 30 22:13:11.303322 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:13:11.303240    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:13:11.749421 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:11.749387    4473 generic.go:356] "Generic (PLEG): container finished" podID="a0887f84-01cf-4b06-aadf-88fb49a5e2ad" containerID="e62a25ef884dd287e32ca5b1d6b028f8c9d47e70b0b6d418d3605558296ef5c9" exitCode=0
Oct 30 22:13:11.749601 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:11.749458    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf" event={"ID":"a0887f84-01cf-4b06-aadf-88fb49a5e2ad","Type":"ContainerDied","Data":"e62a25ef884dd287e32ca5b1d6b028f8c9d47e70b0b6d418d3605558296ef5c9"}
Oct 30 22:13:12.756123 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:12.756035    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf" event={"ID":"a0887f84-01cf-4b06-aadf-88fb49a5e2ad","Type":"ContainerStarted","Data":"b1e5ec11cd40ad695ecc4af62e0892c32f0244a8176d2975dbac433257768e85"}
Oct 30 22:13:12.756123 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:12.756109    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf" event={"ID":"a0887f84-01cf-4b06-aadf-88fb49a5e2ad","Type":"ContainerStarted","Data":"066446615bd62eb52dabb4afc062870db6068e2ff7d70c3c894b7d4b05cd7c21"}
Oct 30 22:13:12.756751 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:12.756322    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:12.878917 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:12.878860    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf" podStartSLOduration=2.947447341 podStartE2EDuration="6.878843917s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.931396576s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:12.877524342 +0000 UTC m=+1211.113733701" watchObservedRunningTime="2026-10-30 22:13:12.878843917 +0000 UTC m=+1211.115053262"
Oct 30 22:13:13.759845 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:13.759812    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:13.761179 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:13.761149    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf" podUID="a0887f84-01cf-4b06-aadf-88fb49a5e2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 30 22:13:14.763410 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:14.763368    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf" podUID="a0887f84-01cf-4b06-aadf-88fb49a5e2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 30 22:13:19.768126 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:19.768093    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:19.769317 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:19.769297    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:26.406249 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:26.406214    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"]
Oct 30 22:13:26.406696 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:26.406435    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf" podUID="a0887f84-01cf-4b06-aadf-88fb49a5e2ad" containerName="kserve-container" containerID="cri-o://066446615bd62eb52dabb4afc062870db6068e2ff7d70c3c894b7d4b05cd7c21" gracePeriod=30
Oct 30 22:13:26.406696 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:26.406535    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf" podUID="a0887f84-01cf-4b06-aadf-88fb49a5e2ad" containerName="kube-rbac-proxy" containerID="cri-o://b1e5ec11cd40ad695ecc4af62e0892c32f0244a8176d2975dbac433257768e85" gracePeriod=30
Oct 30 22:13:26.446964 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:26.446922    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:26.821982 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:26.821889    4473 generic.go:356] "Generic (PLEG): container finished" podID="a0887f84-01cf-4b06-aadf-88fb49a5e2ad" containerID="b1e5ec11cd40ad695ecc4af62e0892c32f0244a8176d2975dbac433257768e85" exitCode=0
Oct 30 22:13:26.822130 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:26.821978    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf" event={"ID":"a0887f84-01cf-4b06-aadf-88fb49a5e2ad","Type":"ContainerDied","Data":"b1e5ec11cd40ad695ecc4af62e0892c32f0244a8176d2975dbac433257768e85"}
Oct 30 22:13:27.054826 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.054793    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:27.126228 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.126145    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-proxy-tls\") pod \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") "
Oct 30 22:13:27.126228 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.126192    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kube-rbac-proxy-sar-config\") pod \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") "
Oct 30 22:13:27.126479 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.126254    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kube-api-access-pvlmc\" (UniqueName: \"kubernetes.io/projected/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kube-api-access-pvlmc\") pod \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") "
Oct 30 22:13:27.126479 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.126291    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kserve-provision-location\") pod \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") "
Oct 30 22:13:27.126604 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.126488    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-cabundle-cert\") pod \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\" (UID: \"a0887f84-01cf-4b06-aadf-88fb49a5e2ad\") "
Oct 30 22:13:27.126674 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.126652    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kserve-provision-location" pod "a0887f84-01cf-4b06-aadf-88fb49a5e2ad" (UID: "a0887f84-01cf-4b06-aadf-88fb49a5e2ad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:27.126729 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.126660    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kube-rbac-proxy-sar-config" pod "a0887f84-01cf-4b06-aadf-88fb49a5e2ad" (UID: "a0887f84-01cf-4b06-aadf-88fb49a5e2ad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:27.126824 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.126810    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:27.126867 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.126827    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:27.126900 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.126852    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-cabundle-cert" pod "a0887f84-01cf-4b06-aadf-88fb49a5e2ad" (UID: "a0887f84-01cf-4b06-aadf-88fb49a5e2ad"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:27.128517 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.128492    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kube-api-access-pvlmc" pod "a0887f84-01cf-4b06-aadf-88fb49a5e2ad" (UID: "a0887f84-01cf-4b06-aadf-88fb49a5e2ad"). InnerVolumeSpecName "kube-api-access-pvlmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:27.128611 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.128574    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-proxy-tls" pod "a0887f84-01cf-4b06-aadf-88fb49a5e2ad" (UID: "a0887f84-01cf-4b06-aadf-88fb49a5e2ad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:27.227595 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.227566    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:27.227595 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.227593    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:27.227595 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.227602    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvlmc\" (UniqueName: \"kubernetes.io/projected/a0887f84-01cf-4b06-aadf-88fb49a5e2ad-kube-api-access-pvlmc\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:27.828436 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.828399    4473 generic.go:356] "Generic (PLEG): container finished" podID="a0887f84-01cf-4b06-aadf-88fb49a5e2ad" containerID="066446615bd62eb52dabb4afc062870db6068e2ff7d70c3c894b7d4b05cd7c21" exitCode=0
Oct 30 22:13:27.828916 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.828446    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf" event={"ID":"a0887f84-01cf-4b06-aadf-88fb49a5e2ad","Type":"ContainerDied","Data":"066446615bd62eb52dabb4afc062870db6068e2ff7d70c3c894b7d4b05cd7c21"}
Oct 30 22:13:27.828916 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.828460    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"
Oct 30 22:13:27.828916 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.828492    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf" event={"ID":"a0887f84-01cf-4b06-aadf-88fb49a5e2ad","Type":"ContainerDied","Data":"26ddb4b50e7073b6456824407155895e6459dfecbb984796d6639d58c99842ce"}
Oct 30 22:13:27.828916 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.828518    4473 scope.go:122] "RemoveContainer" containerID="b1e5ec11cd40ad695ecc4af62e0892c32f0244a8176d2975dbac433257768e85"
Oct 30 22:13:27.837386 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.837365    4473 scope.go:122] "RemoveContainer" containerID="066446615bd62eb52dabb4afc062870db6068e2ff7d70c3c894b7d4b05cd7c21"
Oct 30 22:13:27.844657 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.844638    4473 scope.go:122] "RemoveContainer" containerID="e62a25ef884dd287e32ca5b1d6b028f8c9d47e70b0b6d418d3605558296ef5c9"
Oct 30 22:13:27.851676 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.851661    4473 scope.go:122] "RemoveContainer" containerID="b1e5ec11cd40ad695ecc4af62e0892c32f0244a8176d2975dbac433257768e85"
Oct 30 22:13:27.851945 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:13:27.851928    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1e5ec11cd40ad695ecc4af62e0892c32f0244a8176d2975dbac433257768e85\": container with ID starting with b1e5ec11cd40ad695ecc4af62e0892c32f0244a8176d2975dbac433257768e85 not found: ID does not exist" containerID="b1e5ec11cd40ad695ecc4af62e0892c32f0244a8176d2975dbac433257768e85"
Oct 30 22:13:27.852013 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.851970    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1e5ec11cd40ad695ecc4af62e0892c32f0244a8176d2975dbac433257768e85"} err="failed to get container status \"b1e5ec11cd40ad695ecc4af62e0892c32f0244a8176d2975dbac433257768e85\": rpc error: code = NotFound desc = could not find container \"b1e5ec11cd40ad695ecc4af62e0892c32f0244a8176d2975dbac433257768e85\": container with ID starting with b1e5ec11cd40ad695ecc4af62e0892c32f0244a8176d2975dbac433257768e85 not found: ID does not exist"
Oct 30 22:13:27.852013 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.851991    4473 scope.go:122] "RemoveContainer" containerID="066446615bd62eb52dabb4afc062870db6068e2ff7d70c3c894b7d4b05cd7c21"
Oct 30 22:13:27.852225 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:13:27.852204    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"066446615bd62eb52dabb4afc062870db6068e2ff7d70c3c894b7d4b05cd7c21\": container with ID starting with 066446615bd62eb52dabb4afc062870db6068e2ff7d70c3c894b7d4b05cd7c21 not found: ID does not exist" containerID="066446615bd62eb52dabb4afc062870db6068e2ff7d70c3c894b7d4b05cd7c21"
Oct 30 22:13:27.852299 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.852236    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066446615bd62eb52dabb4afc062870db6068e2ff7d70c3c894b7d4b05cd7c21"} err="failed to get container status \"066446615bd62eb52dabb4afc062870db6068e2ff7d70c3c894b7d4b05cd7c21\": rpc error: code = NotFound desc = could not find container \"066446615bd62eb52dabb4afc062870db6068e2ff7d70c3c894b7d4b05cd7c21\": container with ID starting with 066446615bd62eb52dabb4afc062870db6068e2ff7d70c3c894b7d4b05cd7c21 not found: ID does not exist"
Oct 30 22:13:27.852299 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.852260    4473 scope.go:122] "RemoveContainer" containerID="e62a25ef884dd287e32ca5b1d6b028f8c9d47e70b0b6d418d3605558296ef5c9"
Oct 30 22:13:27.852520 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:13:27.852502    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e62a25ef884dd287e32ca5b1d6b028f8c9d47e70b0b6d418d3605558296ef5c9\": container with ID starting with e62a25ef884dd287e32ca5b1d6b028f8c9d47e70b0b6d418d3605558296ef5c9 not found: ID does not exist" containerID="e62a25ef884dd287e32ca5b1d6b028f8c9d47e70b0b6d418d3605558296ef5c9"
Oct 30 22:13:27.852557 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:27.852526    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e62a25ef884dd287e32ca5b1d6b028f8c9d47e70b0b6d418d3605558296ef5c9"} err="failed to get container status \"e62a25ef884dd287e32ca5b1d6b028f8c9d47e70b0b6d418d3605558296ef5c9\": rpc error: code = NotFound desc = could not find container \"e62a25ef884dd287e32ca5b1d6b028f8c9d47e70b0b6d418d3605558296ef5c9\": container with ID starting with e62a25ef884dd287e32ca5b1d6b028f8c9d47e70b0b6d418d3605558296ef5c9 not found: ID does not exist"
Oct 30 22:13:28.047403 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:28.047370    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"]
Oct 30 22:13:28.053267 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:28.053247    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-lightgbm-predictor-db87765bc-29jrf"]
Oct 30 22:13:28.180731 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:28.180700    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:28.352327 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:28.352292    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:29.022315 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:29.022285    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:29.252826 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:29.252795    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:29.886178 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:29.886147    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0887f84-01cf-4b06-aadf-88fb49a5e2ad" path="/var/lib/kubelet/pods/a0887f84-01cf-4b06-aadf-88fb49a5e2ad/volumes"
Oct 30 22:13:31.097600 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.097563    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:31.324764 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.324723    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:31.602250 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.602210    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"]
Oct 30 22:13:31.602641 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.602623    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a0887f84-01cf-4b06-aadf-88fb49a5e2ad" containerName="kube-rbac-proxy"
Oct 30 22:13:31.602736 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.602643    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0887f84-01cf-4b06-aadf-88fb49a5e2ad" containerName="kube-rbac-proxy"
Oct 30 22:13:31.602736 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.602667    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a0887f84-01cf-4b06-aadf-88fb49a5e2ad" containerName="storage-initializer"
Oct 30 22:13:31.602736 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.602674    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0887f84-01cf-4b06-aadf-88fb49a5e2ad" containerName="storage-initializer"
Oct 30 22:13:31.602736 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.602705    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a0887f84-01cf-4b06-aadf-88fb49a5e2ad" containerName="kserve-container"
Oct 30 22:13:31.602736 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.602712    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0887f84-01cf-4b06-aadf-88fb49a5e2ad" containerName="kserve-container"
Oct 30 22:13:31.603038 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.602785    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0887f84-01cf-4b06-aadf-88fb49a5e2ad" containerName="kserve-container"
Oct 30 22:13:31.603038 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.602798    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0887f84-01cf-4b06-aadf-88fb49a5e2ad" containerName="kube-rbac-proxy"
Oct 30 22:13:31.607466 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.607444    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:31.660663 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.660632    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:13:31.660663 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.660641    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Oct 30 22:13:31.661969 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.661938    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a406c862-25e4-4c0c-a9ed-ca5695072999-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:31.662020 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.661996    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfnn7\" (UniqueName: \"kubernetes.io/projected/a406c862-25e4-4c0c-a9ed-ca5695072999-kube-api-access-vfnn7\") pod \"isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:31.662104 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.662092    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a406c862-25e4-4c0c-a9ed-ca5695072999-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:31.662147 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.662121    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a406c862-25e4-4c0c-a9ed-ca5695072999-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:31.662177 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.662142    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a406c862-25e4-4c0c-a9ed-ca5695072999-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:31.711010 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.710982    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:13:31.711192 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.711020    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:13:31.711269 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.711222    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:13:31.711332 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.711270    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Oct 30 22:13:31.711332 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.711305    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:13:31.712821 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.712799    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"]
Oct 30 22:13:31.763089 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.763056    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a406c862-25e4-4c0c-a9ed-ca5695072999-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:31.763089 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.763089    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a406c862-25e4-4c0c-a9ed-ca5695072999-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:31.763325 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.763112    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a406c862-25e4-4c0c-a9ed-ca5695072999-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:31.763325 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.763135    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a406c862-25e4-4c0c-a9ed-ca5695072999-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:31.763325 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.763174    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfnn7\" (UniqueName: \"kubernetes.io/projected/a406c862-25e4-4c0c-a9ed-ca5695072999-kube-api-access-vfnn7\") pod \"isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:31.763526 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.763504    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a406c862-25e4-4c0c-a9ed-ca5695072999-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:31.763807 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.763786    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a406c862-25e4-4c0c-a9ed-ca5695072999-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:31.763889 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.763845    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a406c862-25e4-4c0c-a9ed-ca5695072999-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:31.765755 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.765736    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a406c862-25e4-4c0c-a9ed-ca5695072999-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:31.821803 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.821771    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfnn7\" (UniqueName: \"kubernetes.io/projected/a406c862-25e4-4c0c-a9ed-ca5695072999-kube-api-access-vfnn7\") pod \"isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") " pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:31.924090 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:31.924064    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:32.348879 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:32.348801    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"]
Oct 30 22:13:32.385873 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:13:32.385847    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda406c862_25e4_4c0c_a9ed_ca5695072999.slice/crio-a1fef4f68684e10880762f8bfab3186ebd676cf145c2a1210aec77cece175d9e WatchSource:0}: Error finding container a1fef4f68684e10880762f8bfab3186ebd676cf145c2a1210aec77cece175d9e: Status 404 returned error can't find the container with id a1fef4f68684e10880762f8bfab3186ebd676cf145c2a1210aec77cece175d9e
Oct 30 22:13:32.856253 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:32.856211    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr" event={"ID":"a406c862-25e4-4c0c-a9ed-ca5695072999","Type":"ContainerStarted","Data":"1bb987addd1d738486260394858efd62b42609cf4eb1ac0fcd17e2a7b98ebdcf"}
Oct 30 22:13:32.856425 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:32.856282    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr" event={"ID":"a406c862-25e4-4c0c-a9ed-ca5695072999","Type":"ContainerStarted","Data":"a1fef4f68684e10880762f8bfab3186ebd676cf145c2a1210aec77cece175d9e"}
Oct 30 22:13:33.303988 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:33.303918    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:33.617939 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:33.617836    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:33.783576 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:33.783512    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:35.845658 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:35.845616    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:35.999616 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:35.999585    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:36.670443 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:36.670410    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:36.874248 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:36.874217    4473 generic.go:356] "Generic (PLEG): container finished" podID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerID="1bb987addd1d738486260394858efd62b42609cf4eb1ac0fcd17e2a7b98ebdcf" exitCode=0
Oct 30 22:13:36.874641 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:36.874288    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr" event={"ID":"a406c862-25e4-4c0c-a9ed-ca5695072999","Type":"ContainerDied","Data":"1bb987addd1d738486260394858efd62b42609cf4eb1ac0fcd17e2a7b98ebdcf"}
Oct 30 22:13:37.880721 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:37.880684    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr" event={"ID":"a406c862-25e4-4c0c-a9ed-ca5695072999","Type":"ContainerStarted","Data":"cd10718c3965bd6b9cb65c6b450a0d01e57251663adde1994acd7badbc094a21"}
Oct 30 22:13:37.885105 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:37.885085    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:37.885240 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:37.885158    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:37.885240 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:37.885178    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr" event={"ID":"a406c862-25e4-4c0c-a9ed-ca5695072999","Type":"ContainerStarted","Data":"9becb2fa0b6ad570345d41809fb1b51b7e283d0a64ecd494d5ac0a690cd611d0"}
Oct 30 22:13:38.053518 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:38.053466    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr" podStartSLOduration=3.4000059990000002 podStartE2EDuration="7.053451807s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.653445808s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:38.052096864 +0000 UTC m=+1236.288306207" watchObservedRunningTime="2026-10-30 22:13:38.053451807 +0000 UTC m=+1236.289661151"
Oct 30 22:13:41.303085 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:13:41.303048    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:13:43.380610 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:43.380580    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:43.538541 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:43.538500    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:43.891012 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:43.890982    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:13:53.437103 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:53.437067    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:58.313082 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:13:58.313049    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:11.304031 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:14:11.303987    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:14:13.892634 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:13.892599    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr" podUID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 22:14:22.808367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:22.808333    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:23.003915 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:23.003877    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:23.892272 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:23.892236    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr" podUID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 22:14:24.134286 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:24.134255    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:33.892174 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:33.892136    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr" podUID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 22:14:41.303446 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:14:41.303410    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:14:44.373487 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:44.373406    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:14:51.616671 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:51.616637    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"]
Oct 30 22:14:51.617096 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:51.616876    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr" podUID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerName="kserve-container" containerID="cri-o://9becb2fa0b6ad570345d41809fb1b51b7e283d0a64ecd494d5ac0a690cd611d0" gracePeriod=30
Oct 30 22:14:51.617096 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:51.616918    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr" podUID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerName="kube-rbac-proxy" containerID="cri-o://cd10718c3965bd6b9cb65c6b450a0d01e57251663adde1994acd7badbc094a21" gracePeriod=30
Oct 30 22:14:51.734577 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:51.734522    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:52.179069 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:52.179030    4473 generic.go:356] "Generic (PLEG): container finished" podID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerID="cd10718c3965bd6b9cb65c6b450a0d01e57251663adde1994acd7badbc094a21" exitCode=0
Oct 30 22:14:52.179246 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:52.179102    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr" event={"ID":"a406c862-25e4-4c0c-a9ed-ca5695072999","Type":"ContainerDied","Data":"cd10718c3965bd6b9cb65c6b450a0d01e57251663adde1994acd7badbc094a21"}
Oct 30 22:14:53.885608 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:53.885565    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr" podUID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Oct 30 22:14:53.892065 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:53.892038    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr" podUID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 22:14:55.644663 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:55.644639    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:55.816158 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:55.816121    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:55.985940 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:55.985853    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:56.459478 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:56.459457    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:14:56.521696 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:56.521623    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a406c862-25e4-4c0c-a9ed-ca5695072999-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a406c862-25e4-4c0c-a9ed-ca5695072999-kube-rbac-proxy-sar-config\") pod \"a406c862-25e4-4c0c-a9ed-ca5695072999\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") "
Oct 30 22:14:56.521696 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:56.521667    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a406c862-25e4-4c0c-a9ed-ca5695072999-kube-api-access-vfnn7\" (UniqueName: \"kubernetes.io/projected/a406c862-25e4-4c0c-a9ed-ca5695072999-kube-api-access-vfnn7\") pod \"a406c862-25e4-4c0c-a9ed-ca5695072999\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") "
Oct 30 22:14:56.521696 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:56.521688    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a406c862-25e4-4c0c-a9ed-ca5695072999-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a406c862-25e4-4c0c-a9ed-ca5695072999-proxy-tls\") pod \"a406c862-25e4-4c0c-a9ed-ca5695072999\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") "
Oct 30 22:14:56.521986 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:56.521730    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a406c862-25e4-4c0c-a9ed-ca5695072999-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a406c862-25e4-4c0c-a9ed-ca5695072999-cabundle-cert\") pod \"a406c862-25e4-4c0c-a9ed-ca5695072999\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") "
Oct 30 22:14:56.521986 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:56.521774    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a406c862-25e4-4c0c-a9ed-ca5695072999-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a406c862-25e4-4c0c-a9ed-ca5695072999-kserve-provision-location\") pod \"a406c862-25e4-4c0c-a9ed-ca5695072999\" (UID: \"a406c862-25e4-4c0c-a9ed-ca5695072999\") "
Oct 30 22:14:56.522121 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:56.522091    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a406c862-25e4-4c0c-a9ed-ca5695072999-kube-rbac-proxy-sar-config" pod "a406c862-25e4-4c0c-a9ed-ca5695072999" (UID: "a406c862-25e4-4c0c-a9ed-ca5695072999"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:56.522182 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:56.522155    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a406c862-25e4-4c0c-a9ed-ca5695072999-cabundle-cert" pod "a406c862-25e4-4c0c-a9ed-ca5695072999" (UID: "a406c862-25e4-4c0c-a9ed-ca5695072999"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:56.522182 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:56.522162    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a406c862-25e4-4c0c-a9ed-ca5695072999-kserve-provision-location" pod "a406c862-25e4-4c0c-a9ed-ca5695072999" (UID: "a406c862-25e4-4c0c-a9ed-ca5695072999"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:56.524002 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:56.523976    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a406c862-25e4-4c0c-a9ed-ca5695072999-proxy-tls" pod "a406c862-25e4-4c0c-a9ed-ca5695072999" (UID: "a406c862-25e4-4c0c-a9ed-ca5695072999"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:56.524101 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:56.524026    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a406c862-25e4-4c0c-a9ed-ca5695072999-kube-api-access-vfnn7" pod "a406c862-25e4-4c0c-a9ed-ca5695072999" (UID: "a406c862-25e4-4c0c-a9ed-ca5695072999"). InnerVolumeSpecName "kube-api-access-vfnn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:56.622891 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:56.622862    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a406c862-25e4-4c0c-a9ed-ca5695072999-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:56.622891 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:56.622887    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a406c862-25e4-4c0c-a9ed-ca5695072999-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:56.622891 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:56.622897    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfnn7\" (UniqueName: \"kubernetes.io/projected/a406c862-25e4-4c0c-a9ed-ca5695072999-kube-api-access-vfnn7\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:56.623134 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:56.622907    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a406c862-25e4-4c0c-a9ed-ca5695072999-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:56.623134 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:56.622916    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a406c862-25e4-4c0c-a9ed-ca5695072999-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:57.202805 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:57.202773    4473 generic.go:356] "Generic (PLEG): container finished" podID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerID="9becb2fa0b6ad570345d41809fb1b51b7e283d0a64ecd494d5ac0a690cd611d0" exitCode=0
Oct 30 22:14:57.203272 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:57.202831    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"
Oct 30 22:14:57.203272 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:57.202839    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr" event={"ID":"a406c862-25e4-4c0c-a9ed-ca5695072999","Type":"ContainerDied","Data":"9becb2fa0b6ad570345d41809fb1b51b7e283d0a64ecd494d5ac0a690cd611d0"}
Oct 30 22:14:57.203272 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:57.202884    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr" event={"ID":"a406c862-25e4-4c0c-a9ed-ca5695072999","Type":"ContainerDied","Data":"a1fef4f68684e10880762f8bfab3186ebd676cf145c2a1210aec77cece175d9e"}
Oct 30 22:14:57.203272 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:57.202907    4473 scope.go:122] "RemoveContainer" containerID="cd10718c3965bd6b9cb65c6b450a0d01e57251663adde1994acd7badbc094a21"
Oct 30 22:14:57.211441 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:57.211422    4473 scope.go:122] "RemoveContainer" containerID="9becb2fa0b6ad570345d41809fb1b51b7e283d0a64ecd494d5ac0a690cd611d0"
Oct 30 22:14:57.218635 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:57.218618    4473 scope.go:122] "RemoveContainer" containerID="1bb987addd1d738486260394858efd62b42609cf4eb1ac0fcd17e2a7b98ebdcf"
Oct 30 22:14:57.225717 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:57.225699    4473 scope.go:122] "RemoveContainer" containerID="cd10718c3965bd6b9cb65c6b450a0d01e57251663adde1994acd7badbc094a21"
Oct 30 22:14:57.225971 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:14:57.225938    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd10718c3965bd6b9cb65c6b450a0d01e57251663adde1994acd7badbc094a21\": container with ID starting with cd10718c3965bd6b9cb65c6b450a0d01e57251663adde1994acd7badbc094a21 not found: ID does not exist" containerID="cd10718c3965bd6b9cb65c6b450a0d01e57251663adde1994acd7badbc094a21"
Oct 30 22:14:57.226036 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:57.225979    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd10718c3965bd6b9cb65c6b450a0d01e57251663adde1994acd7badbc094a21"} err="failed to get container status \"cd10718c3965bd6b9cb65c6b450a0d01e57251663adde1994acd7badbc094a21\": rpc error: code = NotFound desc = could not find container \"cd10718c3965bd6b9cb65c6b450a0d01e57251663adde1994acd7badbc094a21\": container with ID starting with cd10718c3965bd6b9cb65c6b450a0d01e57251663adde1994acd7badbc094a21 not found: ID does not exist"
Oct 30 22:14:57.226036 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:57.225996    4473 scope.go:122] "RemoveContainer" containerID="9becb2fa0b6ad570345d41809fb1b51b7e283d0a64ecd494d5ac0a690cd611d0"
Oct 30 22:14:57.226229 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:14:57.226210    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9becb2fa0b6ad570345d41809fb1b51b7e283d0a64ecd494d5ac0a690cd611d0\": container with ID starting with 9becb2fa0b6ad570345d41809fb1b51b7e283d0a64ecd494d5ac0a690cd611d0 not found: ID does not exist" containerID="9becb2fa0b6ad570345d41809fb1b51b7e283d0a64ecd494d5ac0a690cd611d0"
Oct 30 22:14:57.226292 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:57.226238    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9becb2fa0b6ad570345d41809fb1b51b7e283d0a64ecd494d5ac0a690cd611d0"} err="failed to get container status \"9becb2fa0b6ad570345d41809fb1b51b7e283d0a64ecd494d5ac0a690cd611d0\": rpc error: code = NotFound desc = could not find container \"9becb2fa0b6ad570345d41809fb1b51b7e283d0a64ecd494d5ac0a690cd611d0\": container with ID starting with 9becb2fa0b6ad570345d41809fb1b51b7e283d0a64ecd494d5ac0a690cd611d0 not found: ID does not exist"
Oct 30 22:14:57.226292 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:57.226261    4473 scope.go:122] "RemoveContainer" containerID="1bb987addd1d738486260394858efd62b42609cf4eb1ac0fcd17e2a7b98ebdcf"
Oct 30 22:14:57.226509 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:14:57.226494    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb987addd1d738486260394858efd62b42609cf4eb1ac0fcd17e2a7b98ebdcf\": container with ID starting with 1bb987addd1d738486260394858efd62b42609cf4eb1ac0fcd17e2a7b98ebdcf not found: ID does not exist" containerID="1bb987addd1d738486260394858efd62b42609cf4eb1ac0fcd17e2a7b98ebdcf"
Oct 30 22:14:57.226552 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:57.226511    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb987addd1d738486260394858efd62b42609cf4eb1ac0fcd17e2a7b98ebdcf"} err="failed to get container status \"1bb987addd1d738486260394858efd62b42609cf4eb1ac0fcd17e2a7b98ebdcf\": rpc error: code = NotFound desc = could not find container \"1bb987addd1d738486260394858efd62b42609cf4eb1ac0fcd17e2a7b98ebdcf\": container with ID starting with 1bb987addd1d738486260394858efd62b42609cf4eb1ac0fcd17e2a7b98ebdcf not found: ID does not exist"
Oct 30 22:14:57.422115 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:57.422082    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"]
Oct 30 22:14:57.473211 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:57.473129    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-sklearn-v2-predictor-b5f64699f-6njmr"]
Oct 30 22:14:57.885636 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:57.885546    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a406c862-25e4-4c0c-a9ed-ca5695072999" path="/var/lib/kubelet/pods/a406c862-25e4-4c0c-a9ed-ca5695072999/volumes"
Oct 30 22:14:58.863229 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:58.863193    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:59.411448 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:59.411409    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:59.686456 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:14:59.686375    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:00.018411 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:00.018296    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:00.403539 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:00.403508    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:01.517484 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:01.517448    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:01.829374 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:01.829297    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:01.962445 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:01.962409    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"]
Oct 30 22:15:01.962828 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:01.962811    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerName="storage-initializer"
Oct 30 22:15:01.962919 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:01.962831    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerName="storage-initializer"
Oct 30 22:15:01.962919 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:01.962854    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerName="kserve-container"
Oct 30 22:15:01.962919 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:01.962862    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerName="kserve-container"
Oct 30 22:15:01.962919 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:01.962880    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerName="kube-rbac-proxy"
Oct 30 22:15:01.962919 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:01.962887    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerName="kube-rbac-proxy"
Oct 30 22:15:01.963202 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:01.962995    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerName="kube-rbac-proxy"
Oct 30 22:15:01.963202 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:01.963015    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a406c862-25e4-4c0c-a9ed-ca5695072999" containerName="kserve-container"
Oct 30 22:15:01.967856 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:01.967835    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:02.022195 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.022175    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:15:02.066993 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.066968    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b575869c-1bf2-47ad-905a-710c7870eb18-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:02.067107 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.067039    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b575869c-1bf2-47ad-905a-710c7870eb18-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:02.067107 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.067081    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b575869c-1bf2-47ad-905a-710c7870eb18-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:02.067107 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.067101    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7qhb\" (UniqueName: \"kubernetes.io/projected/b575869c-1bf2-47ad-905a-710c7870eb18-kube-api-access-w7qhb\") pod \"isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:02.067207 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.067126    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b575869c-1bf2-47ad-905a-710c7870eb18-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:02.073128 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.073103    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:15:02.073248 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.073190    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:15:02.073248 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.073228    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:15:02.073248 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.073228    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Oct 30 22:15:02.073535 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.073228    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Oct 30 22:15:02.073535 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.073229    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:15:02.076347 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.076326    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"]
Oct 30 22:15:02.168091 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.168063    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b575869c-1bf2-47ad-905a-710c7870eb18-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:02.168241 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.168118    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b575869c-1bf2-47ad-905a-710c7870eb18-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:02.168241 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.168141    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b575869c-1bf2-47ad-905a-710c7870eb18-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:02.168241 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.168166    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b575869c-1bf2-47ad-905a-710c7870eb18-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:02.168241 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.168184    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7qhb\" (UniqueName: \"kubernetes.io/projected/b575869c-1bf2-47ad-905a-710c7870eb18-kube-api-access-w7qhb\") pod \"isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:02.168622 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.168601    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b575869c-1bf2-47ad-905a-710c7870eb18-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:02.168750 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.168731    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b575869c-1bf2-47ad-905a-710c7870eb18-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:02.168852 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.168833    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b575869c-1bf2-47ad-905a-710c7870eb18-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:02.170688 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.170672    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b575869c-1bf2-47ad-905a-710c7870eb18-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:02.229936 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.229911    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7qhb\" (UniqueName: \"kubernetes.io/projected/b575869c-1bf2-47ad-905a-710c7870eb18-kube-api-access-w7qhb\") pod \"isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") " pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:02.284499 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.284464    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:02.349859 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.349829    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:02.708451 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.708425    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"]
Oct 30 22:15:02.711133 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:15:02.711106    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb575869c_1bf2_47ad_905a_710c7870eb18.slice/crio-eaea085bd3396bcf592c432197543ebe7ca40583cd067445e294d1897adba870 WatchSource:0}: Error finding container eaea085bd3396bcf592c432197543ebe7ca40583cd067445e294d1897adba870: Status 404 returned error can't find the container with id eaea085bd3396bcf592c432197543ebe7ca40583cd067445e294d1897adba870
Oct 30 22:15:02.785232 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:02.785204    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:03.106892 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:03.106813    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:03.239553 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:03.239515    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv" event={"ID":"b575869c-1bf2-47ad-905a-710c7870eb18","Type":"ContainerStarted","Data":"b19aded25326d8c12b0ca62d48e68804fa90644a580f1fe2888f999821945ce4"}
Oct 30 22:15:03.239724 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:03.239628    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv" event={"ID":"b575869c-1bf2-47ad-905a-710c7870eb18","Type":"ContainerStarted","Data":"eaea085bd3396bcf592c432197543ebe7ca40583cd067445e294d1897adba870"}
Oct 30 22:15:07.256942 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:07.256911    4473 generic.go:356] "Generic (PLEG): container finished" podID="b575869c-1bf2-47ad-905a-710c7870eb18" containerID="b19aded25326d8c12b0ca62d48e68804fa90644a580f1fe2888f999821945ce4" exitCode=0
Oct 30 22:15:07.257460 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:07.256990    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv" event={"ID":"b575869c-1bf2-47ad-905a-710c7870eb18","Type":"ContainerDied","Data":"b19aded25326d8c12b0ca62d48e68804fa90644a580f1fe2888f999821945ce4"}
Oct 30 22:15:08.263396 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:08.263357    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv" event={"ID":"b575869c-1bf2-47ad-905a-710c7870eb18","Type":"ContainerStarted","Data":"a6ff7df5168ead5a3c94c495c91bdb7bc75e78f1b2397e2a91e3f13cc611e154"}
Oct 30 22:15:08.263812 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:08.263419    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv" event={"ID":"b575869c-1bf2-47ad-905a-710c7870eb18","Type":"ContainerStarted","Data":"8a415de2ec9807e0f2e646d7d4da7b307958b971992afc928225d387dd629c05"}
Oct 30 22:15:08.263812 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:08.263739    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:08.263812 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:08.263805    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:08.436186 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:08.436123    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv" podStartSLOduration=3.522869287 podStartE2EDuration="7.436104473s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.913235186s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:08.43299071 +0000 UTC m=+1326.669200064" watchObservedRunningTime="2026-10-30 22:15:08.436104473 +0000 UTC m=+1326.672313819"
Oct 30 22:15:10.946344 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:10.946317    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:11.120814 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:11.120782    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:11.303876 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:15:11.303791    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:15:14.276272 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:14.276244    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:15:41.303662 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:15:41.303624    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:15:44.277585 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:44.277550    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv" podUID="b575869c-1bf2-47ad-905a-710c7870eb18" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 22:15:54.277168 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:54.277128    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv" podUID="b575869c-1bf2-47ad-905a-710c7870eb18" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 22:15:58.864693 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:58.864657    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:59.026611 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:59.026579    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:59.959202 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:15:59.959156    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:04.277312 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:04.277274    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv" podUID="b575869c-1bf2-47ad-905a-710c7870eb18" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 22:16:11.305631 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:16:11.305587    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:16:14.864934 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:14.864848    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:16:20.730352 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:20.730304    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"]
Oct 30 22:16:20.730930 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:20.730544    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv" podUID="b575869c-1bf2-47ad-905a-710c7870eb18" containerName="kserve-container" containerID="cri-o://8a415de2ec9807e0f2e646d7d4da7b307958b971992afc928225d387dd629c05" gracePeriod=30
Oct 30 22:16:20.730930 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:20.730581    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv" podUID="b575869c-1bf2-47ad-905a-710c7870eb18" containerName="kube-rbac-proxy" containerID="cri-o://a6ff7df5168ead5a3c94c495c91bdb7bc75e78f1b2397e2a91e3f13cc611e154" gracePeriod=30
Oct 30 22:16:20.752791 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:20.752769    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:21.558905 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:21.558870    4473 generic.go:356] "Generic (PLEG): container finished" podID="b575869c-1bf2-47ad-905a-710c7870eb18" containerID="a6ff7df5168ead5a3c94c495c91bdb7bc75e78f1b2397e2a91e3f13cc611e154" exitCode=0
Oct 30 22:16:21.559097 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:21.558963    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv" event={"ID":"b575869c-1bf2-47ad-905a-710c7870eb18","Type":"ContainerDied","Data":"a6ff7df5168ead5a3c94c495c91bdb7bc75e78f1b2397e2a91e3f13cc611e154"}
Oct 30 22:16:22.476290 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:22.476258    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:24.271189 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:24.271141    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv" podUID="b575869c-1bf2-47ad-905a-710c7870eb18" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Oct 30 22:16:24.277612 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:24.277585    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv" podUID="b575869c-1bf2-47ad-905a-710c7870eb18" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 22:16:25.373575 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.373552    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:16:25.527814 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.527738    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b575869c-1bf2-47ad-905a-710c7870eb18-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b575869c-1bf2-47ad-905a-710c7870eb18-kube-rbac-proxy-sar-config\") pod \"b575869c-1bf2-47ad-905a-710c7870eb18\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") "
Oct 30 22:16:25.527814 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.527774    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b575869c-1bf2-47ad-905a-710c7870eb18-proxy-tls\" (UniqueName: \"kubernetes.io/secret/b575869c-1bf2-47ad-905a-710c7870eb18-proxy-tls\") pod \"b575869c-1bf2-47ad-905a-710c7870eb18\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") "
Oct 30 22:16:25.527814 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.527801    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b575869c-1bf2-47ad-905a-710c7870eb18-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b575869c-1bf2-47ad-905a-710c7870eb18-kserve-provision-location\") pod \"b575869c-1bf2-47ad-905a-710c7870eb18\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") "
Oct 30 22:16:25.528057 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.527839    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b575869c-1bf2-47ad-905a-710c7870eb18-kube-api-access-w7qhb\" (UniqueName: \"kubernetes.io/projected/b575869c-1bf2-47ad-905a-710c7870eb18-kube-api-access-w7qhb\") pod \"b575869c-1bf2-47ad-905a-710c7870eb18\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") "
Oct 30 22:16:25.528057 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.527914    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b575869c-1bf2-47ad-905a-710c7870eb18-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b575869c-1bf2-47ad-905a-710c7870eb18-cabundle-cert\") pod \"b575869c-1bf2-47ad-905a-710c7870eb18\" (UID: \"b575869c-1bf2-47ad-905a-710c7870eb18\") "
Oct 30 22:16:25.528229 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.528201    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b575869c-1bf2-47ad-905a-710c7870eb18-kube-rbac-proxy-sar-config" pod "b575869c-1bf2-47ad-905a-710c7870eb18" (UID: "b575869c-1bf2-47ad-905a-710c7870eb18"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:25.528286 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.528231    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b575869c-1bf2-47ad-905a-710c7870eb18-kserve-provision-location" pod "b575869c-1bf2-47ad-905a-710c7870eb18" (UID: "b575869c-1bf2-47ad-905a-710c7870eb18"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:25.528326 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.528283    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b575869c-1bf2-47ad-905a-710c7870eb18-cabundle-cert" pod "b575869c-1bf2-47ad-905a-710c7870eb18" (UID: "b575869c-1bf2-47ad-905a-710c7870eb18"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:25.530067 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.530044    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b575869c-1bf2-47ad-905a-710c7870eb18-proxy-tls" pod "b575869c-1bf2-47ad-905a-710c7870eb18" (UID: "b575869c-1bf2-47ad-905a-710c7870eb18"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:25.530181 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.530161    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b575869c-1bf2-47ad-905a-710c7870eb18-kube-api-access-w7qhb" pod "b575869c-1bf2-47ad-905a-710c7870eb18" (UID: "b575869c-1bf2-47ad-905a-710c7870eb18"). InnerVolumeSpecName "kube-api-access-w7qhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:25.576906 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.576876    4473 generic.go:356] "Generic (PLEG): container finished" podID="b575869c-1bf2-47ad-905a-710c7870eb18" containerID="8a415de2ec9807e0f2e646d7d4da7b307958b971992afc928225d387dd629c05" exitCode=0
Oct 30 22:16:25.577151 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.576937    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv" event={"ID":"b575869c-1bf2-47ad-905a-710c7870eb18","Type":"ContainerDied","Data":"8a415de2ec9807e0f2e646d7d4da7b307958b971992afc928225d387dd629c05"}
Oct 30 22:16:25.577151 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.576940    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"
Oct 30 22:16:25.577151 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.577008    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv" event={"ID":"b575869c-1bf2-47ad-905a-710c7870eb18","Type":"ContainerDied","Data":"eaea085bd3396bcf592c432197543ebe7ca40583cd067445e294d1897adba870"}
Oct 30 22:16:25.577151 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.577034    4473 scope.go:122] "RemoveContainer" containerID="a6ff7df5168ead5a3c94c495c91bdb7bc75e78f1b2397e2a91e3f13cc611e154"
Oct 30 22:16:25.585429 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.585410    4473 scope.go:122] "RemoveContainer" containerID="8a415de2ec9807e0f2e646d7d4da7b307958b971992afc928225d387dd629c05"
Oct 30 22:16:25.592930 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.592911    4473 scope.go:122] "RemoveContainer" containerID="b19aded25326d8c12b0ca62d48e68804fa90644a580f1fe2888f999821945ce4"
Oct 30 22:16:25.599986 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.599969    4473 scope.go:122] "RemoveContainer" containerID="a6ff7df5168ead5a3c94c495c91bdb7bc75e78f1b2397e2a91e3f13cc611e154"
Oct 30 22:16:25.600223 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:16:25.600206    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ff7df5168ead5a3c94c495c91bdb7bc75e78f1b2397e2a91e3f13cc611e154\": container with ID starting with a6ff7df5168ead5a3c94c495c91bdb7bc75e78f1b2397e2a91e3f13cc611e154 not found: ID does not exist" containerID="a6ff7df5168ead5a3c94c495c91bdb7bc75e78f1b2397e2a91e3f13cc611e154"
Oct 30 22:16:25.600275 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.600235    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ff7df5168ead5a3c94c495c91bdb7bc75e78f1b2397e2a91e3f13cc611e154"} err="failed to get container status \"a6ff7df5168ead5a3c94c495c91bdb7bc75e78f1b2397e2a91e3f13cc611e154\": rpc error: code = NotFound desc = could not find container \"a6ff7df5168ead5a3c94c495c91bdb7bc75e78f1b2397e2a91e3f13cc611e154\": container with ID starting with a6ff7df5168ead5a3c94c495c91bdb7bc75e78f1b2397e2a91e3f13cc611e154 not found: ID does not exist"
Oct 30 22:16:25.600275 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.600252    4473 scope.go:122] "RemoveContainer" containerID="8a415de2ec9807e0f2e646d7d4da7b307958b971992afc928225d387dd629c05"
Oct 30 22:16:25.600514 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:16:25.600499    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a415de2ec9807e0f2e646d7d4da7b307958b971992afc928225d387dd629c05\": container with ID starting with 8a415de2ec9807e0f2e646d7d4da7b307958b971992afc928225d387dd629c05 not found: ID does not exist" containerID="8a415de2ec9807e0f2e646d7d4da7b307958b971992afc928225d387dd629c05"
Oct 30 22:16:25.600566 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.600520    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a415de2ec9807e0f2e646d7d4da7b307958b971992afc928225d387dd629c05"} err="failed to get container status \"8a415de2ec9807e0f2e646d7d4da7b307958b971992afc928225d387dd629c05\": rpc error: code = NotFound desc = could not find container \"8a415de2ec9807e0f2e646d7d4da7b307958b971992afc928225d387dd629c05\": container with ID starting with 8a415de2ec9807e0f2e646d7d4da7b307958b971992afc928225d387dd629c05 not found: ID does not exist"
Oct 30 22:16:25.600566 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.600532    4473 scope.go:122] "RemoveContainer" containerID="b19aded25326d8c12b0ca62d48e68804fa90644a580f1fe2888f999821945ce4"
Oct 30 22:16:25.600779 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:16:25.600762    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b19aded25326d8c12b0ca62d48e68804fa90644a580f1fe2888f999821945ce4\": container with ID starting with b19aded25326d8c12b0ca62d48e68804fa90644a580f1fe2888f999821945ce4 not found: ID does not exist" containerID="b19aded25326d8c12b0ca62d48e68804fa90644a580f1fe2888f999821945ce4"
Oct 30 22:16:25.600826 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.600784    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b19aded25326d8c12b0ca62d48e68804fa90644a580f1fe2888f999821945ce4"} err="failed to get container status \"b19aded25326d8c12b0ca62d48e68804fa90644a580f1fe2888f999821945ce4\": rpc error: code = NotFound desc = could not find container \"b19aded25326d8c12b0ca62d48e68804fa90644a580f1fe2888f999821945ce4\": container with ID starting with b19aded25326d8c12b0ca62d48e68804fa90644a580f1fe2888f999821945ce4 not found: ID does not exist"
Oct 30 22:16:25.628440 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.628421    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b575869c-1bf2-47ad-905a-710c7870eb18-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:25.628530 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.628441    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b575869c-1bf2-47ad-905a-710c7870eb18-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:25.628530 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.628452    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b575869c-1bf2-47ad-905a-710c7870eb18-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:25.628530 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.628461    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b575869c-1bf2-47ad-905a-710c7870eb18-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:25.628530 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.628471    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7qhb\" (UniqueName: \"kubernetes.io/projected/b575869c-1bf2-47ad-905a-710c7870eb18-kube-api-access-w7qhb\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:25.742471 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.742440    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"]
Oct 30 22:16:25.748229 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.748204    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-xgboost-v2-predictor-86b447f864-r7bbv"]
Oct 30 22:16:25.886475 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:25.886402    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b575869c-1bf2-47ad-905a-710c7870eb18" path="/var/lib/kubelet/pods/b575869c-1bf2-47ad-905a-710c7870eb18/volumes"
Oct 30 22:16:29.773729 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.773695    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"]
Oct 30 22:16:29.774155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.774031    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b575869c-1bf2-47ad-905a-710c7870eb18" containerName="storage-initializer"
Oct 30 22:16:29.774155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.774042    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b575869c-1bf2-47ad-905a-710c7870eb18" containerName="storage-initializer"
Oct 30 22:16:29.774155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.774049    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b575869c-1bf2-47ad-905a-710c7870eb18" containerName="kube-rbac-proxy"
Oct 30 22:16:29.774155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.774055    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b575869c-1bf2-47ad-905a-710c7870eb18" containerName="kube-rbac-proxy"
Oct 30 22:16:29.774155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.774080    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b575869c-1bf2-47ad-905a-710c7870eb18" containerName="kserve-container"
Oct 30 22:16:29.774155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.774085    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b575869c-1bf2-47ad-905a-710c7870eb18" containerName="kserve-container"
Oct 30 22:16:29.774155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.774133    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b575869c-1bf2-47ad-905a-710c7870eb18" containerName="kserve-container"
Oct 30 22:16:29.774155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.774142    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b575869c-1bf2-47ad-905a-710c7870eb18" containerName="kube-rbac-proxy"
Oct 30 22:16:29.778782 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.778763    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:29.832385 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.832356    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:16:29.865835 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.865806    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c18b7e3d-3e77-484d-9386-51cbff434966-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:29.866030 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.865853    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c18b7e3d-3e77-484d-9386-51cbff434966-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:29.866030 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.865932    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c18b7e3d-3e77-484d-9386-51cbff434966-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:29.866030 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.865986    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqvpk\" (UniqueName: \"kubernetes.io/projected/c18b7e3d-3e77-484d-9386-51cbff434966-kube-api-access-jqvpk\") pod \"isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:29.866138 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.866082    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c18b7e3d-3e77-484d-9386-51cbff434966-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:29.882694 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.882657    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:16:29.882694 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.882657    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Oct 30 22:16:29.882901 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.882731    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:16:29.882901 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.882761    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:16:29.882901 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.882783    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:16:29.883212 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.883189    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Oct 30 22:16:29.887494 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.887471    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"]
Oct 30 22:16:29.967262 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.967235    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c18b7e3d-3e77-484d-9386-51cbff434966-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:29.967262 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.967273    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c18b7e3d-3e77-484d-9386-51cbff434966-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:29.967489 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.967370    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c18b7e3d-3e77-484d-9386-51cbff434966-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:29.967489 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.967422    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c18b7e3d-3e77-484d-9386-51cbff434966-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:29.967489 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.967445    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvpk\" (UniqueName: \"kubernetes.io/projected/c18b7e3d-3e77-484d-9386-51cbff434966-kube-api-access-jqvpk\") pod \"isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:29.967929 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.967897    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c18b7e3d-3e77-484d-9386-51cbff434966-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:29.968050 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.968026    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c18b7e3d-3e77-484d-9386-51cbff434966-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:29.968104 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.968088    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c18b7e3d-3e77-484d-9386-51cbff434966-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:29.970139 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:29.970122    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c18b7e3d-3e77-484d-9386-51cbff434966-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:30.024783 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:30.024717    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvpk\" (UniqueName: \"kubernetes.io/projected/c18b7e3d-3e77-484d-9386-51cbff434966-kube-api-access-jqvpk\") pod \"isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") " pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:30.095928 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:30.095887    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:30.518171 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:30.518148    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"]
Oct 30 22:16:30.520356 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:16:30.520326    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18b7e3d_3e77_484d_9386_51cbff434966.slice/crio-7a2021cbbee0423e1e7dd9440d884abd0e3b2af303110be5256ff382ead91890 WatchSource:0}: Error finding container 7a2021cbbee0423e1e7dd9440d884abd0e3b2af303110be5256ff382ead91890: Status 404 returned error can't find the container with id 7a2021cbbee0423e1e7dd9440d884abd0e3b2af303110be5256ff382ead91890
Oct 30 22:16:30.597691 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:30.597663    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq" event={"ID":"c18b7e3d-3e77-484d-9386-51cbff434966","Type":"ContainerStarted","Data":"0192c52f9ba6c8ba2b00dcc4fa877ee81ef2b60ca0927ff39acf236347cb96e7"}
Oct 30 22:16:30.597820 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:30.597718    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq" event={"ID":"c18b7e3d-3e77-484d-9386-51cbff434966","Type":"ContainerStarted","Data":"7a2021cbbee0423e1e7dd9440d884abd0e3b2af303110be5256ff382ead91890"}
Oct 30 22:16:30.656814 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:30.656790    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:30.809314 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:30.809208    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:30.966386 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:30.966350    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:32.883212 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:32.883163    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:33.040501 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:33.040466    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:35.618275 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:35.618245    4473 generic.go:356] "Generic (PLEG): container finished" podID="c18b7e3d-3e77-484d-9386-51cbff434966" containerID="0192c52f9ba6c8ba2b00dcc4fa877ee81ef2b60ca0927ff39acf236347cb96e7" exitCode=0
Oct 30 22:16:35.618681 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:35.618321    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq" event={"ID":"c18b7e3d-3e77-484d-9386-51cbff434966","Type":"ContainerDied","Data":"0192c52f9ba6c8ba2b00dcc4fa877ee81ef2b60ca0927ff39acf236347cb96e7"}
Oct 30 22:16:36.557741 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:36.557701    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:36.625358 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:36.625324    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq" event={"ID":"c18b7e3d-3e77-484d-9386-51cbff434966","Type":"ContainerStarted","Data":"64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e"}
Oct 30 22:16:36.625822 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:36.625382    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq" event={"ID":"c18b7e3d-3e77-484d-9386-51cbff434966","Type":"ContainerStarted","Data":"fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f"}
Oct 30 22:16:36.625822 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:36.625619    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:36.625822 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:36.625786    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:36.749716 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:36.749655    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq" podStartSLOduration=3.527277302 podStartE2EDuration="7.749641884s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.222364582s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:36.747871551 +0000 UTC m=+1414.984080897" watchObservedRunningTime="2026-10-30 22:16:36.749641884 +0000 UTC m=+1414.985851229"
Oct 30 22:16:36.906397 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:36.906364    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:37.070000 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:37.069964    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:39.138850 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:39.138816    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:39.308201 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:39.308171    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:39.992842 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:39.992807    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:41.304077 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:16:41.304037    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:16:42.636884 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:42.636852    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:16:46.189461 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:46.189423    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:46.364216 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:46.364163    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:59.331633 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:16:59.331597    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:11.304244 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:17:11.304205    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:17:12.696727 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:12.696690    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:17:19.877577 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:19.877538    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"]
Oct 30 22:17:19.878062 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:19.877786    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq" podUID="c18b7e3d-3e77-484d-9386-51cbff434966" containerName="kserve-container" containerID="cri-o://fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f" gracePeriod=30
Oct 30 22:17:19.878062 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:19.877824    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq" podUID="c18b7e3d-3e77-484d-9386-51cbff434966" containerName="kube-rbac-proxy" containerID="cri-o://64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e" gracePeriod=30
Oct 30 22:17:19.908754 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:19.908726    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:20.520225 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.520201    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:17:20.551336 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.551310    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18b7e3d-3e77-484d-9386-51cbff434966-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c18b7e3d-3e77-484d-9386-51cbff434966-kube-rbac-proxy-sar-config\") pod \"c18b7e3d-3e77-484d-9386-51cbff434966\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") "
Oct 30 22:17:20.551485 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.551363    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c18b7e3d-3e77-484d-9386-51cbff434966-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c18b7e3d-3e77-484d-9386-51cbff434966-proxy-tls\") pod \"c18b7e3d-3e77-484d-9386-51cbff434966\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") "
Oct 30 22:17:20.551485 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.551396    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18b7e3d-3e77-484d-9386-51cbff434966-kube-api-access-jqvpk\" (UniqueName: \"kubernetes.io/projected/c18b7e3d-3e77-484d-9386-51cbff434966-kube-api-access-jqvpk\") pod \"c18b7e3d-3e77-484d-9386-51cbff434966\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") "
Oct 30 22:17:20.551485 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.551465    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18b7e3d-3e77-484d-9386-51cbff434966-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c18b7e3d-3e77-484d-9386-51cbff434966-kserve-provision-location\") pod \"c18b7e3d-3e77-484d-9386-51cbff434966\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") "
Oct 30 22:17:20.551644 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.551523    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18b7e3d-3e77-484d-9386-51cbff434966-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c18b7e3d-3e77-484d-9386-51cbff434966-cabundle-cert\") pod \"c18b7e3d-3e77-484d-9386-51cbff434966\" (UID: \"c18b7e3d-3e77-484d-9386-51cbff434966\") "
Oct 30 22:17:20.551848 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.551717    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18b7e3d-3e77-484d-9386-51cbff434966-kube-rbac-proxy-sar-config" pod "c18b7e3d-3e77-484d-9386-51cbff434966" (UID: "c18b7e3d-3e77-484d-9386-51cbff434966"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:20.552000 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.551851    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18b7e3d-3e77-484d-9386-51cbff434966-kserve-provision-location" pod "c18b7e3d-3e77-484d-9386-51cbff434966" (UID: "c18b7e3d-3e77-484d-9386-51cbff434966"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:20.552000 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.551901    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18b7e3d-3e77-484d-9386-51cbff434966-cabundle-cert" pod "c18b7e3d-3e77-484d-9386-51cbff434966" (UID: "c18b7e3d-3e77-484d-9386-51cbff434966"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:20.553734 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.553709    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c18b7e3d-3e77-484d-9386-51cbff434966-proxy-tls" pod "c18b7e3d-3e77-484d-9386-51cbff434966" (UID: "c18b7e3d-3e77-484d-9386-51cbff434966"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:20.553834 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.553733    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18b7e3d-3e77-484d-9386-51cbff434966-kube-api-access-jqvpk" pod "c18b7e3d-3e77-484d-9386-51cbff434966" (UID: "c18b7e3d-3e77-484d-9386-51cbff434966"). InnerVolumeSpecName "kube-api-access-jqvpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:20.652898 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.652872    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c18b7e3d-3e77-484d-9386-51cbff434966-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:20.652898 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.652899    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c18b7e3d-3e77-484d-9386-51cbff434966-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:20.653102 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.652909    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqvpk\" (UniqueName: \"kubernetes.io/projected/c18b7e3d-3e77-484d-9386-51cbff434966-kube-api-access-jqvpk\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:20.653102 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.652919    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c18b7e3d-3e77-484d-9386-51cbff434966-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:20.653102 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.652928    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c18b7e3d-3e77-484d-9386-51cbff434966-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:20.809345 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.809314    4473 generic.go:356] "Generic (PLEG): container finished" podID="c18b7e3d-3e77-484d-9386-51cbff434966" containerID="64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e" exitCode=0
Oct 30 22:17:20.809345 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.809338    4473 generic.go:356] "Generic (PLEG): container finished" podID="c18b7e3d-3e77-484d-9386-51cbff434966" containerID="fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f" exitCode=0
Oct 30 22:17:20.809553 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.809368    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq" event={"ID":"c18b7e3d-3e77-484d-9386-51cbff434966","Type":"ContainerDied","Data":"64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e"}
Oct 30 22:17:20.809553 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.809379    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"
Oct 30 22:17:20.809553 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.809415    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq" event={"ID":"c18b7e3d-3e77-484d-9386-51cbff434966","Type":"ContainerDied","Data":"fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f"}
Oct 30 22:17:20.809553 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.809436    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq" event={"ID":"c18b7e3d-3e77-484d-9386-51cbff434966","Type":"ContainerDied","Data":"7a2021cbbee0423e1e7dd9440d884abd0e3b2af303110be5256ff382ead91890"}
Oct 30 22:17:20.809553 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.809457    4473 scope.go:122] "RemoveContainer" containerID="64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e"
Oct 30 22:17:20.818162 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.818146    4473 scope.go:122] "RemoveContainer" containerID="fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f"
Oct 30 22:17:20.825682 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.825666    4473 scope.go:122] "RemoveContainer" containerID="0192c52f9ba6c8ba2b00dcc4fa877ee81ef2b60ca0927ff39acf236347cb96e7"
Oct 30 22:17:20.832448 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.832432    4473 scope.go:122] "RemoveContainer" containerID="64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e"
Oct 30 22:17:20.832720 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:17:20.832701    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e\": container with ID starting with 64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e not found: ID does not exist" containerID="64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e"
Oct 30 22:17:20.832775 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.832728    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e"} err="failed to get container status \"64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e\": rpc error: code = NotFound desc = could not find container \"64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e\": container with ID starting with 64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e not found: ID does not exist"
Oct 30 22:17:20.832775 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.832746    4473 scope.go:122] "RemoveContainer" containerID="fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f"
Oct 30 22:17:20.832999 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:17:20.832982    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f\": container with ID starting with fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f not found: ID does not exist" containerID="fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f"
Oct 30 22:17:20.833055 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.833007    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f"} err="failed to get container status \"fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f\": rpc error: code = NotFound desc = could not find container \"fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f\": container with ID starting with fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f not found: ID does not exist"
Oct 30 22:17:20.833055 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.833021    4473 scope.go:122] "RemoveContainer" containerID="0192c52f9ba6c8ba2b00dcc4fa877ee81ef2b60ca0927ff39acf236347cb96e7"
Oct 30 22:17:20.833250 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:17:20.833234    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0192c52f9ba6c8ba2b00dcc4fa877ee81ef2b60ca0927ff39acf236347cb96e7\": container with ID starting with 0192c52f9ba6c8ba2b00dcc4fa877ee81ef2b60ca0927ff39acf236347cb96e7 not found: ID does not exist" containerID="0192c52f9ba6c8ba2b00dcc4fa877ee81ef2b60ca0927ff39acf236347cb96e7"
Oct 30 22:17:20.833288 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.833257    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0192c52f9ba6c8ba2b00dcc4fa877ee81ef2b60ca0927ff39acf236347cb96e7"} err="failed to get container status \"0192c52f9ba6c8ba2b00dcc4fa877ee81ef2b60ca0927ff39acf236347cb96e7\": rpc error: code = NotFound desc = could not find container \"0192c52f9ba6c8ba2b00dcc4fa877ee81ef2b60ca0927ff39acf236347cb96e7\": container with ID starting with 0192c52f9ba6c8ba2b00dcc4fa877ee81ef2b60ca0927ff39acf236347cb96e7 not found: ID does not exist"
Oct 30 22:17:20.833288 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.833270    4473 scope.go:122] "RemoveContainer" containerID="64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e"
Oct 30 22:17:20.833489 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.833473    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e"} err="failed to get container status \"64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e\": rpc error: code = NotFound desc = could not find container \"64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e\": container with ID starting with 64314ae412944b3bfc1079c5fbe7799fdcb565f8ec823ebef2365f416836cf2e not found: ID does not exist"
Oct 30 22:17:20.833527 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.833489    4473 scope.go:122] "RemoveContainer" containerID="fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f"
Oct 30 22:17:20.833706 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.833689    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f"} err="failed to get container status \"fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f\": rpc error: code = NotFound desc = could not find container \"fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f\": container with ID starting with fb54d87e72dd816b1fb551b8cb9018a4fe77481080dd80f4f3bf1963bb10217f not found: ID does not exist"
Oct 30 22:17:20.833762 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.833706    4473 scope.go:122] "RemoveContainer" containerID="0192c52f9ba6c8ba2b00dcc4fa877ee81ef2b60ca0927ff39acf236347cb96e7"
Oct 30 22:17:20.833897 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.833883    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0192c52f9ba6c8ba2b00dcc4fa877ee81ef2b60ca0927ff39acf236347cb96e7"} err="failed to get container status \"0192c52f9ba6c8ba2b00dcc4fa877ee81ef2b60ca0927ff39acf236347cb96e7\": rpc error: code = NotFound desc = could not find container \"0192c52f9ba6c8ba2b00dcc4fa877ee81ef2b60ca0927ff39acf236347cb96e7\": container with ID starting with 0192c52f9ba6c8ba2b00dcc4fa877ee81ef2b60ca0927ff39acf236347cb96e7 not found: ID does not exist"
Oct 30 22:17:20.974467 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.974393    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"]
Oct 30 22:17:20.978255 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:20.978234    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-predictive-lightgbm-v2-predictor-6b5bd6f78f-nnjpq"]
Oct 30 22:17:21.891822 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:21.891787    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18b7e3d-3e77-484d-9386-51cbff434966" path="/var/lib/kubelet/pods/c18b7e3d-3e77-484d-9386-51cbff434966/volumes"
Oct 30 22:17:24.417465 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:24.417422    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:24.641168 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:24.641136    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:26.718323 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:26.718291    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:26.903337 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:26.903305    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:38.044898 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.044864    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:38.358864 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.358781    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:38.543812 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.543779    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"]
Oct 30 22:17:38.544118 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.544106    4473 cpu_manager.go:414] "containerMap: removing container" podUID="c18b7e3d-3e77-484d-9386-51cbff434966" containerName="storage-initializer"
Oct 30 22:17:38.544164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.544119    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18b7e3d-3e77-484d-9386-51cbff434966" containerName="storage-initializer"
Oct 30 22:17:38.544164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.544133    4473 cpu_manager.go:414] "containerMap: removing container" podUID="c18b7e3d-3e77-484d-9386-51cbff434966" containerName="kube-rbac-proxy"
Oct 30 22:17:38.544164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.544138    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18b7e3d-3e77-484d-9386-51cbff434966" containerName="kube-rbac-proxy"
Oct 30 22:17:38.544164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.544154    4473 cpu_manager.go:414] "containerMap: removing container" podUID="c18b7e3d-3e77-484d-9386-51cbff434966" containerName="kserve-container"
Oct 30 22:17:38.544164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.544160    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18b7e3d-3e77-484d-9386-51cbff434966" containerName="kserve-container"
Oct 30 22:17:38.544321 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.544222    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18b7e3d-3e77-484d-9386-51cbff434966" containerName="kube-rbac-proxy"
Oct 30 22:17:38.544321 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.544232    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18b7e3d-3e77-484d-9386-51cbff434966" containerName="kserve-container"
Oct 30 22:17:38.548758 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.548740    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:38.656404 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.656371    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Oct 30 22:17:38.656404 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.656377    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:17:38.656649 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.656462    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:17:38.656649 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.656469    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:17:38.656649 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.656476    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:17:38.656649 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.656464    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:17:38.656649 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.656610    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-predictor-serving-cert\""
Oct 30 22:17:38.658080 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.658056    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"]
Oct 30 22:17:38.699935 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.699910    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9044c2e8-b764-46da-af8a-28e6fc020370-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-87f646997-wv6dt\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") " pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:38.700108 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.699965    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9044c2e8-b764-46da-af8a-28e6fc020370-proxy-tls\") pod \"isvc-sklearn-predictor-87f646997-wv6dt\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") " pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:38.700108 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.699985    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9044c2e8-b764-46da-af8a-28e6fc020370-kserve-provision-location\") pod \"isvc-sklearn-predictor-87f646997-wv6dt\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") " pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:38.700108 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.700041    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs2x8\" (UniqueName: \"kubernetes.io/projected/9044c2e8-b764-46da-af8a-28e6fc020370-kube-api-access-bs2x8\") pod \"isvc-sklearn-predictor-87f646997-wv6dt\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") " pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:38.700229 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.700130    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9044c2e8-b764-46da-af8a-28e6fc020370-cabundle-cert\") pod \"isvc-sklearn-predictor-87f646997-wv6dt\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") " pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:38.801448 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.801415    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9044c2e8-b764-46da-af8a-28e6fc020370-cabundle-cert\") pod \"isvc-sklearn-predictor-87f646997-wv6dt\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") " pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:38.801637 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.801464    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9044c2e8-b764-46da-af8a-28e6fc020370-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-87f646997-wv6dt\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") " pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:38.801637 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.801581    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9044c2e8-b764-46da-af8a-28e6fc020370-proxy-tls\") pod \"isvc-sklearn-predictor-87f646997-wv6dt\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") " pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:38.801637 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.801612    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9044c2e8-b764-46da-af8a-28e6fc020370-kserve-provision-location\") pod \"isvc-sklearn-predictor-87f646997-wv6dt\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") " pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:38.801801 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.801654    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs2x8\" (UniqueName: \"kubernetes.io/projected/9044c2e8-b764-46da-af8a-28e6fc020370-kube-api-access-bs2x8\") pod \"isvc-sklearn-predictor-87f646997-wv6dt\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") " pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:38.802017 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.801993    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9044c2e8-b764-46da-af8a-28e6fc020370-kserve-provision-location\") pod \"isvc-sklearn-predictor-87f646997-wv6dt\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") " pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:38.802292 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.802272    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9044c2e8-b764-46da-af8a-28e6fc020370-cabundle-cert\") pod \"isvc-sklearn-predictor-87f646997-wv6dt\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") " pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:38.802339 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.802272    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9044c2e8-b764-46da-af8a-28e6fc020370-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-87f646997-wv6dt\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") " pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:38.804214 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.804192    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9044c2e8-b764-46da-af8a-28e6fc020370-proxy-tls\") pod \"isvc-sklearn-predictor-87f646997-wv6dt\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") " pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:38.861273 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.861243    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs2x8\" (UniqueName: \"kubernetes.io/projected/9044c2e8-b764-46da-af8a-28e6fc020370-kube-api-access-bs2x8\") pod \"isvc-sklearn-predictor-87f646997-wv6dt\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") " pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:38.865068 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:38.865051    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:39.290101 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:39.290072    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"]
Oct 30 22:17:39.292893 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:17:39.292854    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9044c2e8_b764_46da_af8a_28e6fc020370.slice/crio-7418fad685aba99a6f4001f0d5d0da61236d25a57083917d69e6c74b7526923a WatchSource:0}: Error finding container 7418fad685aba99a6f4001f0d5d0da61236d25a57083917d69e6c74b7526923a: Status 404 returned error can't find the container with id 7418fad685aba99a6f4001f0d5d0da61236d25a57083917d69e6c74b7526923a
Oct 30 22:17:39.889289 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:39.889249    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" event={"ID":"9044c2e8-b764-46da-af8a-28e6fc020370","Type":"ContainerStarted","Data":"aabbc5bb9fc842385031afba43fa2f4caba711975b2ce03f1b1c1269c2fd52d8"}
Oct 30 22:17:39.889469 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:39.889327    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" event={"ID":"9044c2e8-b764-46da-af8a-28e6fc020370","Type":"ContainerStarted","Data":"7418fad685aba99a6f4001f0d5d0da61236d25a57083917d69e6c74b7526923a"}
Oct 30 22:17:41.303375 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:17:41.303338    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:17:43.906149 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:43.906114    4473 generic.go:356] "Generic (PLEG): container finished" podID="9044c2e8-b764-46da-af8a-28e6fc020370" containerID="aabbc5bb9fc842385031afba43fa2f4caba711975b2ce03f1b1c1269c2fd52d8" exitCode=0
Oct 30 22:17:43.906587 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:43.906189    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" event={"ID":"9044c2e8-b764-46da-af8a-28e6fc020370","Type":"ContainerDied","Data":"aabbc5bb9fc842385031afba43fa2f4caba711975b2ce03f1b1c1269c2fd52d8"}
Oct 30 22:17:44.912762 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:44.912726    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" event={"ID":"9044c2e8-b764-46da-af8a-28e6fc020370","Type":"ContainerStarted","Data":"aee1d859fa85a9d52eb75fb5cd6b9e9bd866b09c744a1f15bae23f7a07d02d04"}
Oct 30 22:17:44.913184 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:44.912787    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" event={"ID":"9044c2e8-b764-46da-af8a-28e6fc020370","Type":"ContainerStarted","Data":"f8e0bd6bce85ea6236b31818610ac19c1d0af99e5111fffaa68150fc995a7bd1"}
Oct 30 22:17:44.913184 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:44.912983    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:45.083926 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:45.083856    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" podStartSLOduration=3.291179162 podStartE2EDuration="7.083838582s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.79265942s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:45.083415308 +0000 UTC m=+1483.319624656" watchObservedRunningTime="2026-10-30 22:17:45.083838582 +0000 UTC m=+1483.320047929"
Oct 30 22:17:45.916793 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:45.916756    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:45.918097 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:45.918076    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 22:17:46.921002 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:46.920926    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 22:17:47.472625 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:47.472580    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:47.628364 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:47.628333    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:48.703232 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:48.703180    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:51.925879 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:51.925853    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:17:51.926320 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:51.926297    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 22:17:53.710306 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:53.710265    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:59.693246 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:17:59.693213    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:01.926550 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:01.926509    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 22:18:02.012043 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:02.012007    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:18:02.012248 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:02.012233    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:18:08.551881 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:08.551842    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:11.303251 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:18:11.303212    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:18:11.926517 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:11.926485    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 22:18:19.452843 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:19.452800    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:19.617261 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:19.617215    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:21.499874 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:21.499838    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:21.729909 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:21.729874    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:21.927281 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:21.927247    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 22:18:25.192249 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:25.192208    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:25.395618 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:25.395587    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:25.710028 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:25.709994    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:25.864324 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:25.864287    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:28.329975 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:28.329928    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:28.487512 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:28.487481    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:28.775190 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:28.775158    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:31.926281 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:31.926244    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 22:18:34.668505 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:34.668468    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:34.876213 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:34.876179    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:41.303442 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:18:41.303401    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:18:41.926972 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:41.926926    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 22:18:51.926689 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:51.926653    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 22:18:56.881614 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:18:56.881573    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 22:19:00.063406 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:00.063371    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:06.882461 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:06.882432    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:19:11.304295 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:19:11.304257    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:19:17.930967 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:17.930856    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-predictor-87f646997-wv6dt_9044c2e8-b764-46da-af8a-28e6fc020370/kserve-container/0.log"
Oct 30 22:19:20.001811 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:20.001783    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-predictor-87f646997-wv6dt_9044c2e8-b764-46da-af8a-28e6fc020370/kserve-container/0.log"
Oct 30 22:19:22.122170 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:22.122137    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-predictor-87f646997-wv6dt_9044c2e8-b764-46da-af8a-28e6fc020370/kserve-container/0.log"
Oct 30 22:19:24.193164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:24.193130    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-predictor-87f646997-wv6dt_9044c2e8-b764-46da-af8a-28e6fc020370/kserve-container/0.log"
Oct 30 22:19:26.264741 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:26.264711    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-predictor-87f646997-wv6dt_9044c2e8-b764-46da-af8a-28e6fc020370/kserve-container/0.log"
Oct 30 22:19:28.336107 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:28.336074    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-predictor-87f646997-wv6dt_9044c2e8-b764-46da-af8a-28e6fc020370/kserve-container/0.log"
Oct 30 22:19:30.457436 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:30.457403    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-predictor-87f646997-wv6dt_9044c2e8-b764-46da-af8a-28e6fc020370/kserve-container/0.log"
Oct 30 22:19:32.529194 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:32.529154    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-predictor-87f646997-wv6dt_9044c2e8-b764-46da-af8a-28e6fc020370/kserve-container/0.log"
Oct 30 22:19:33.371131 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:33.371097    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:34.648614 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:34.648589    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-predictor-87f646997-wv6dt_9044c2e8-b764-46da-af8a-28e6fc020370/kserve-container/0.log"
Oct 30 22:19:36.719606 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:36.719560    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-predictor-87f646997-wv6dt_9044c2e8-b764-46da-af8a-28e6fc020370/kserve-container/0.log"
Oct 30 22:19:38.791267 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:38.791237    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-predictor-87f646997-wv6dt_9044c2e8-b764-46da-af8a-28e6fc020370/kserve-container/0.log"
Oct 30 22:19:40.910875 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:40.910847    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-predictor-87f646997-wv6dt_9044c2e8-b764-46da-af8a-28e6fc020370/kserve-container/0.log"
Oct 30 22:19:41.303823 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:19:41.303734    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:19:42.982720 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:42.982688    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-predictor-87f646997-wv6dt_9044c2e8-b764-46da-af8a-28e6fc020370/kserve-container/0.log"
Oct 30 22:19:45.059667 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:45.059631    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-predictor-87f646997-wv6dt_9044c2e8-b764-46da-af8a-28e6fc020370/kserve-container/0.log"
Oct 30 22:19:47.131446 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:47.131411    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-predictor-87f646997-wv6dt_9044c2e8-b764-46da-af8a-28e6fc020370/kserve-container/0.log"
Oct 30 22:19:49.252136 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:49.252093    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-predictor-87f646997-wv6dt_9044c2e8-b764-46da-af8a-28e6fc020370/kserve-container/0.log"
Oct 30 22:19:50.613902 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:50.613871    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"]
Oct 30 22:19:50.614359 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:50.614136    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kserve-container" containerID="cri-o://f8e0bd6bce85ea6236b31818610ac19c1d0af99e5111fffaa68150fc995a7bd1" gracePeriod=30
Oct 30 22:19:50.614359 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:50.614183    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kube-rbac-proxy" containerID="cri-o://aee1d859fa85a9d52eb75fb5cd6b9e9bd866b09c744a1f15bae23f7a07d02d04" gracePeriod=30
Oct 30 22:19:50.631914 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:50.631889    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:51.448220 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:51.448185    4473 generic.go:356] "Generic (PLEG): container finished" podID="9044c2e8-b764-46da-af8a-28e6fc020370" containerID="aee1d859fa85a9d52eb75fb5cd6b9e9bd866b09c744a1f15bae23f7a07d02d04" exitCode=0
Oct 30 22:19:51.448396 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:51.448261    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" event={"ID":"9044c2e8-b764-46da-af8a-28e6fc020370","Type":"ContainerDied","Data":"aee1d859fa85a9d52eb75fb5cd6b9e9bd866b09c744a1f15bae23f7a07d02d04"}
Oct 30 22:19:51.922008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:51.921975    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Oct 30 22:19:55.468514 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.468483    4473 generic.go:356] "Generic (PLEG): container finished" podID="9044c2e8-b764-46da-af8a-28e6fc020370" containerID="f8e0bd6bce85ea6236b31818610ac19c1d0af99e5111fffaa68150fc995a7bd1" exitCode=0
Oct 30 22:19:55.469150 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.468523    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" event={"ID":"9044c2e8-b764-46da-af8a-28e6fc020370","Type":"ContainerDied","Data":"f8e0bd6bce85ea6236b31818610ac19c1d0af99e5111fffaa68150fc995a7bd1"}
Oct 30 22:19:55.559340 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.559315    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:19:55.627028 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.626993    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9044c2e8-b764-46da-af8a-28e6fc020370-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9044c2e8-b764-46da-af8a-28e6fc020370-cabundle-cert\") pod \"9044c2e8-b764-46da-af8a-28e6fc020370\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") "
Oct 30 22:19:55.627188 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.627059    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9044c2e8-b764-46da-af8a-28e6fc020370-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9044c2e8-b764-46da-af8a-28e6fc020370-kserve-provision-location\") pod \"9044c2e8-b764-46da-af8a-28e6fc020370\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") "
Oct 30 22:19:55.627188 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.627083    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9044c2e8-b764-46da-af8a-28e6fc020370-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9044c2e8-b764-46da-af8a-28e6fc020370-proxy-tls\") pod \"9044c2e8-b764-46da-af8a-28e6fc020370\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") "
Oct 30 22:19:55.627188 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.627111    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9044c2e8-b764-46da-af8a-28e6fc020370-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9044c2e8-b764-46da-af8a-28e6fc020370-kube-rbac-proxy-sar-config\") pod \"9044c2e8-b764-46da-af8a-28e6fc020370\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") "
Oct 30 22:19:55.627188 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.627150    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9044c2e8-b764-46da-af8a-28e6fc020370-kube-api-access-bs2x8\" (UniqueName: \"kubernetes.io/projected/9044c2e8-b764-46da-af8a-28e6fc020370-kube-api-access-bs2x8\") pod \"9044c2e8-b764-46da-af8a-28e6fc020370\" (UID: \"9044c2e8-b764-46da-af8a-28e6fc020370\") "
Oct 30 22:19:55.627432 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.627403    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9044c2e8-b764-46da-af8a-28e6fc020370-kserve-provision-location" pod "9044c2e8-b764-46da-af8a-28e6fc020370" (UID: "9044c2e8-b764-46da-af8a-28e6fc020370"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:55.627432 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.627426    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9044c2e8-b764-46da-af8a-28e6fc020370-cabundle-cert" pod "9044c2e8-b764-46da-af8a-28e6fc020370" (UID: "9044c2e8-b764-46da-af8a-28e6fc020370"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:55.627544 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.627474    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9044c2e8-b764-46da-af8a-28e6fc020370-kube-rbac-proxy-sar-config" pod "9044c2e8-b764-46da-af8a-28e6fc020370" (UID: "9044c2e8-b764-46da-af8a-28e6fc020370"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:55.629377 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.629355    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9044c2e8-b764-46da-af8a-28e6fc020370-kube-api-access-bs2x8" pod "9044c2e8-b764-46da-af8a-28e6fc020370" (UID: "9044c2e8-b764-46da-af8a-28e6fc020370"). InnerVolumeSpecName "kube-api-access-bs2x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:55.629461 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.629441    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9044c2e8-b764-46da-af8a-28e6fc020370-proxy-tls" pod "9044c2e8-b764-46da-af8a-28e6fc020370" (UID: "9044c2e8-b764-46da-af8a-28e6fc020370"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:55.728419 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.728330    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9044c2e8-b764-46da-af8a-28e6fc020370-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:55.728419 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.728365    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9044c2e8-b764-46da-af8a-28e6fc020370-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:55.728419 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.728375    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9044c2e8-b764-46da-af8a-28e6fc020370-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:55.728419 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.728384    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs2x8\" (UniqueName: \"kubernetes.io/projected/9044c2e8-b764-46da-af8a-28e6fc020370-kube-api-access-bs2x8\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:55.728419 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:55.728394    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9044c2e8-b764-46da-af8a-28e6fc020370-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:56.474652 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:56.474625    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"
Oct 30 22:19:56.475122 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:56.474623    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt" event={"ID":"9044c2e8-b764-46da-af8a-28e6fc020370","Type":"ContainerDied","Data":"7418fad685aba99a6f4001f0d5d0da61236d25a57083917d69e6c74b7526923a"}
Oct 30 22:19:56.475122 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:56.474761    4473 scope.go:122] "RemoveContainer" containerID="aee1d859fa85a9d52eb75fb5cd6b9e9bd866b09c744a1f15bae23f7a07d02d04"
Oct 30 22:19:56.482962 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:56.482933    4473 scope.go:122] "RemoveContainer" containerID="f8e0bd6bce85ea6236b31818610ac19c1d0af99e5111fffaa68150fc995a7bd1"
Oct 30 22:19:56.489939 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:56.489925    4473 scope.go:122] "RemoveContainer" containerID="aabbc5bb9fc842385031afba43fa2f4caba711975b2ce03f1b1c1269c2fd52d8"
Oct 30 22:19:56.692750 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:56.692720    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"]
Oct 30 22:19:56.695995 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:56.695974    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-predictor-87f646997-wv6dt"]
Oct 30 22:19:57.886195 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:57.886164    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" path="/var/lib/kubelet/pods/9044c2e8-b764-46da-af8a-28e6fc020370/volumes"
Oct 30 22:19:59.400269 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.400230    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:59.641885 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.641859    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:59.865386 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.865305    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"]
Oct 30 22:19:59.865610 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.865599    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kube-rbac-proxy"
Oct 30 22:19:59.865653 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.865612    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kube-rbac-proxy"
Oct 30 22:19:59.865653 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.865632    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kserve-container"
Oct 30 22:19:59.865653 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.865637    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kserve-container"
Oct 30 22:19:59.865653 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.865647    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="storage-initializer"
Oct 30 22:19:59.865653 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.865652    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="storage-initializer"
Oct 30 22:19:59.865812 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.865700    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kube-rbac-proxy"
Oct 30 22:19:59.865812 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.865711    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9044c2e8-b764-46da-af8a-28e6fc020370" containerName="kserve-container"
Oct 30 22:19:59.870199 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.870183    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:19:59.935348 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.935318    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:19:59.935348 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.935332    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:19:59.962772 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.962748    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jrs8\" (UniqueName: \"kubernetes.io/projected/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kube-api-access-5jrs8\") pod \"sklearn-v2-mlserver-predictor-6569c8f4df-khkqv\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") " pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:19:59.962841 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.962778    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-6569c8f4df-khkqv\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") " pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:19:59.962841 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.962807    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-6569c8f4df-khkqv\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") " pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:19:59.962928 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.962912    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-6569c8f4df-khkqv\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") " pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:19:59.963002 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.962987    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-6569c8f4df-khkqv\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") " pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:19:59.981368 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.981346    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 30 22:19:59.981368 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.981353    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:19:59.981368 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.981359    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:19:59.981603 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.981384    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:19:59.981603 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.981407    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Oct 30 22:19:59.983251 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:19:59.983229    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"]
Oct 30 22:20:00.063524 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:00.063493    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-6569c8f4df-khkqv\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") " pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:20:00.063710 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:00.063543    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-6569c8f4df-khkqv\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") " pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:20:00.063710 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:00.063566    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-6569c8f4df-khkqv\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") " pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:20:00.063710 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:00.063608    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jrs8\" (UniqueName: \"kubernetes.io/projected/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kube-api-access-5jrs8\") pod \"sklearn-v2-mlserver-predictor-6569c8f4df-khkqv\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") " pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:20:00.063710 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:00.063629    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-6569c8f4df-khkqv\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") " pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:20:00.064124 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:00.064106    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-6569c8f4df-khkqv\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") " pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:20:00.064317 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:00.064298    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-6569c8f4df-khkqv\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") " pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:20:00.064375 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:00.064355    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-6569c8f4df-khkqv\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") " pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:20:00.066304 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:00.066287    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-6569c8f4df-khkqv\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") " pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:20:00.121984 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:00.121898    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jrs8\" (UniqueName: \"kubernetes.io/projected/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kube-api-access-5jrs8\") pod \"sklearn-v2-mlserver-predictor-6569c8f4df-khkqv\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") " pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:20:00.187001 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:00.186943    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:20:00.346535 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:00.346496    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:00.611629 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:00.611087    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"]
Oct 30 22:20:00.616787 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:20:00.616758    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13da6d9_72cd_4c78_893f_6db0a8b1b9fe.slice/crio-e3adcb7fb2541850396c186a5b8e2a41e230c9a7e9166979ffda742fe28b26a9 WatchSource:0}: Error finding container e3adcb7fb2541850396c186a5b8e2a41e230c9a7e9166979ffda742fe28b26a9: Status 404 returned error can't find the container with id e3adcb7fb2541850396c186a5b8e2a41e230c9a7e9166979ffda742fe28b26a9
Oct 30 22:20:01.495823 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:01.495784    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv" event={"ID":"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe","Type":"ContainerStarted","Data":"803cdf9344bbf90504157167f941e59b30dbe761e58160fe272dc167f5ab8ab2"}
Oct 30 22:20:01.496067 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:01.495849    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv" event={"ID":"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe","Type":"ContainerStarted","Data":"e3adcb7fb2541850396c186a5b8e2a41e230c9a7e9166979ffda742fe28b26a9"}
Oct 30 22:20:04.508840 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:04.508810    4473 generic.go:356] "Generic (PLEG): container finished" podID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" containerID="803cdf9344bbf90504157167f941e59b30dbe761e58160fe272dc167f5ab8ab2" exitCode=0
Oct 30 22:20:04.509164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:04.508852    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv" event={"ID":"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe","Type":"ContainerDied","Data":"803cdf9344bbf90504157167f941e59b30dbe761e58160fe272dc167f5ab8ab2"}
Oct 30 22:20:04.510019 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:04.510002    4473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:20:11.311079 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:20:11.311043    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:20:20.273423 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:20.273386    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:20.426867 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:20.426832    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:21.357728 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:21.357695    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:41.311165 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:20:41.311120    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:20:52.160276 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:52.160181    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:52.460468 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:52.460375    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:55.803773 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:55.803728    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:55.973612 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:20:55.973573    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:00.685399 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:00.685348    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:07.384293 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:07.384248    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:07.585551 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:07.585515    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:11.305992 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:21:11.305939    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:21:16.032543 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.032505    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:16.409204 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.409170    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:16.689751 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.689669    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"]
Oct 30 22:21:16.724658 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.724625    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:16.778161 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.778129    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Oct 30 22:21:16.778161 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.778129    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:21:16.799691 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.799662    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-proxy-tls\") pod \"isvc-logger-predictor-746864dcdc-9wnjk\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") " pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:16.799884 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.799712    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2v5b\" (UniqueName: \"kubernetes.io/projected/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kube-api-access-r2v5b\") pod \"isvc-logger-predictor-746864dcdc-9wnjk\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") " pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:16.799884 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.799743    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kserve-provision-location\") pod \"isvc-logger-predictor-746864dcdc-9wnjk\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") " pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:16.799884 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.799871    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-cabundle-cert\") pod \"isvc-logger-predictor-746864dcdc-9wnjk\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") " pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:16.800089 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.799897    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-746864dcdc-9wnjk\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") " pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:16.828355 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.828314    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-logger-predictor-serving-cert\""
Oct 30 22:21:16.828513 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.828370    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"kube-root-ca.crt\""
Oct 30 22:21:16.828513 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.828408    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"openshift-service-ca.crt\""
Oct 30 22:21:16.828513 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.828443    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"seaweedfs-s3-creds\""
Oct 30 22:21:16.828649 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.828612    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"default-dockercfg-4v4s8\""
Oct 30 22:21:16.830152 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.830130    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"]
Oct 30 22:21:16.900891 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.900861    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-cabundle-cert\") pod \"isvc-logger-predictor-746864dcdc-9wnjk\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") " pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:16.901099 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.900904    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-746864dcdc-9wnjk\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") " pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:16.901099 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.901006    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-proxy-tls\") pod \"isvc-logger-predictor-746864dcdc-9wnjk\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") " pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:16.901099 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.901065    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2v5b\" (UniqueName: \"kubernetes.io/projected/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kube-api-access-r2v5b\") pod \"isvc-logger-predictor-746864dcdc-9wnjk\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") " pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:16.901272 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.901100    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kserve-provision-location\") pod \"isvc-logger-predictor-746864dcdc-9wnjk\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") " pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:16.901517 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.901491    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kserve-provision-location\") pod \"isvc-logger-predictor-746864dcdc-9wnjk\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") " pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:16.901662 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.901644    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-746864dcdc-9wnjk\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") " pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:16.901733 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.901641    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-cabundle-cert\") pod \"isvc-logger-predictor-746864dcdc-9wnjk\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") " pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:16.906459 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.906437    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-proxy-tls\") pod \"isvc-logger-predictor-746864dcdc-9wnjk\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") " pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:16.960170 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:16.960100    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2v5b\" (UniqueName: \"kubernetes.io/projected/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kube-api-access-r2v5b\") pod \"isvc-logger-predictor-746864dcdc-9wnjk\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") " pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:17.041281 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:17.041245    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:17.476796 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:17.476773    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"]
Oct 30 22:21:17.480166 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:21:17.480137    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5f7e4e_ead8_4b99_b829_b1bde61f8868.slice/crio-2014ba29df32a1d75ffde2ce7b92e9336265a7b91d8311ebd7084b53b8d84096 WatchSource:0}: Error finding container 2014ba29df32a1d75ffde2ce7b92e9336265a7b91d8311ebd7084b53b8d84096: Status 404 returned error can't find the container with id 2014ba29df32a1d75ffde2ce7b92e9336265a7b91d8311ebd7084b53b8d84096
Oct 30 22:21:17.862579 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:17.862497    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" event={"ID":"dd5f7e4e-ead8-4b99-b829-b1bde61f8868","Type":"ContainerStarted","Data":"1faf4e10340541fd88b5b8c726502605938c921e325a90888dad1e9c6c662ba2"}
Oct 30 22:21:17.862743 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:17.862584    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" event={"ID":"dd5f7e4e-ead8-4b99-b829-b1bde61f8868","Type":"ContainerStarted","Data":"2014ba29df32a1d75ffde2ce7b92e9336265a7b91d8311ebd7084b53b8d84096"}
Oct 30 22:21:21.883275 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:21.883236    4473 generic.go:356] "Generic (PLEG): container finished" podID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerID="1faf4e10340541fd88b5b8c726502605938c921e325a90888dad1e9c6c662ba2" exitCode=0
Oct 30 22:21:21.886615 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:21.886582    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" event={"ID":"dd5f7e4e-ead8-4b99-b829-b1bde61f8868","Type":"ContainerDied","Data":"1faf4e10340541fd88b5b8c726502605938c921e325a90888dad1e9c6c662ba2"}
Oct 30 22:21:22.891518 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:22.891482    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" event={"ID":"dd5f7e4e-ead8-4b99-b829-b1bde61f8868","Type":"ContainerStarted","Data":"201b24217875af62872779cdf40830c1e09dc23bd77b9a6f17a4631b1e90042e"}
Oct 30 22:21:22.891945 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:22.891543    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" event={"ID":"dd5f7e4e-ead8-4b99-b829-b1bde61f8868","Type":"ContainerStarted","Data":"2c2cc95ef0a6bcfef1b52ea719239672391c9ebcf220f66ed4f792c718b3878c"}
Oct 30 22:21:22.891945 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:22.891569    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" event={"ID":"dd5f7e4e-ead8-4b99-b829-b1bde61f8868","Type":"ContainerStarted","Data":"4dbde38f4fe3e87950daf48e7742cf078482b84fd1069d0bc3bba4994498a9a2"}
Oct 30 22:21:22.892093 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:22.892034    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:22.892093 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:22.892085    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:22.892191 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:22.892109    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:22.893750 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:22.893713    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 22:21:22.894501 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:22.894473    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:21:23.014737 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:23.014672    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podStartSLOduration=3.21338283 podStartE2EDuration="7.014651268s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.801268438s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:23.013469342 +0000 UTC m=+1701.249678687" watchObservedRunningTime="2026-10-30 22:21:23.014651268 +0000 UTC m=+1701.250860618"
Oct 30 22:21:23.900811 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:23.900764    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 22:21:23.901423 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:23.901400    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:21:28.903846 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:28.903817    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:21:28.904574 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:28.904535    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 22:21:28.904908 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:28.904889    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:21:38.905249 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:38.905211    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 22:21:38.905898 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:38.905870    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:21:41.307562 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:21:41.307516    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:21:48.905165 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:48.905121    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 22:21:48.905799 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:48.905677    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:21:58.904925 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:58.904880    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 22:21:58.905505 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:21:58.905482    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:22:01.044266 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:01.044233    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:05.098960 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:05.098912    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv" event={"ID":"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe","Type":"ContainerStarted","Data":"aaadfc88902b171c4d8f10f2e2a74e75995464443dec03be1fe8323a3fd9dd99"}
Oct 30 22:22:05.099367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:05.099023    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:22:05.099367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:05.099176    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:22:05.099367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:05.099223    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv" event={"ID":"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe","Type":"ContainerStarted","Data":"98842152522d0612fafe0d9d450b8e7252506240bca7473b1ebaae82ecbd1ab1"}
Oct 30 22:22:05.270835 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:05.270753    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv" podStartSLOduration=2.7942161690000002 podStartE2EDuration="2m6.270734235s" totalImagesPullingTime="1m59.689096384s" totalInitContainerRuntime="3.787421682s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:05.270059864 +0000 UTC m=+1743.506269201" watchObservedRunningTime="2026-10-30 22:22:05.270734235 +0000 UTC m=+1743.506943656"
Oct 30 22:22:08.904701 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:08.904654    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 22:22:08.905303 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:08.905102    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:22:11.108796 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:11.108768    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:22:11.366341 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:22:11.366258    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:22:18.904990 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:18.904872    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 22:22:18.905479 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:18.905329    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:22:28.905403 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:28.905353    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 22:22:28.905856 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:28.905832    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:22:38.904430 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:38.904377    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 22:22:38.904920 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:38.904836    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:22:41.165793 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:41.165745    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv" podUID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 30 22:22:41.304198 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:22:41.304151    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:22:48.905142 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:48.905115    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:22:48.906312 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:48.905427    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:22:51.112032 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:22:51.111995    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:23:00.502333 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:00.502297    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"]
Oct 30 22:23:00.502744 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:00.502614    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv" podUID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" containerName="kserve-container" containerID="cri-o://98842152522d0612fafe0d9d450b8e7252506240bca7473b1ebaae82ecbd1ab1" gracePeriod=30
Oct 30 22:23:00.502744 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:00.502639    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv" podUID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" containerName="kube-rbac-proxy" containerID="cri-o://aaadfc88902b171c4d8f10f2e2a74e75995464443dec03be1fe8323a3fd9dd99" gracePeriod=30
Oct 30 22:23:00.531772 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:00.531742    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:01.103784 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:01.103738    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv" podUID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Oct 30 22:23:01.336785 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:01.336753    4473 generic.go:356] "Generic (PLEG): container finished" podID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" containerID="aaadfc88902b171c4d8f10f2e2a74e75995464443dec03be1fe8323a3fd9dd99" exitCode=0
Oct 30 22:23:01.336994 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:01.336826    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv" event={"ID":"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe","Type":"ContainerDied","Data":"aaadfc88902b171c4d8f10f2e2a74e75995464443dec03be1fe8323a3fd9dd99"}
Oct 30 22:23:01.597308 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:01.597267    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:02.041001 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:02.040971    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:23:02.041181 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:02.040971    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:23:02.151113 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:02.151077    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv" podUID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.41:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.622570 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:02.622530    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"]
Oct 30 22:23:02.622991 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:02.622778    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kserve-container" containerID="cri-o://4dbde38f4fe3e87950daf48e7742cf078482b84fd1069d0bc3bba4994498a9a2" gracePeriod=30
Oct 30 22:23:02.622991 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:02.622789    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="agent" containerID="cri-o://201b24217875af62872779cdf40830c1e09dc23bd77b9a6f17a4631b1e90042e" gracePeriod=30
Oct 30 22:23:02.622991 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:02.622814    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kube-rbac-proxy" containerID="cri-o://2c2cc95ef0a6bcfef1b52ea719239672391c9ebcf220f66ed4f792c718b3878c" gracePeriod=30
Oct 30 22:23:02.636363 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:02.636340    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:02.945690 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:02.945661    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:03.346858 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:03.346779    4473 generic.go:356] "Generic (PLEG): container finished" podID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerID="2c2cc95ef0a6bcfef1b52ea719239672391c9ebcf220f66ed4f792c718b3878c" exitCode=0
Oct 30 22:23:03.347035 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:03.346852    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" event={"ID":"dd5f7e4e-ead8-4b99-b829-b1bde61f8868","Type":"ContainerDied","Data":"2c2cc95ef0a6bcfef1b52ea719239672391c9ebcf220f66ed4f792c718b3878c"}
Oct 30 22:23:03.899154 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:03.899114    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Oct 30 22:23:06.104333 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:06.104297    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv" podUID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Oct 30 22:23:08.197126 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.197096    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:23:08.247116 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.247096    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kube-api-access-5jrs8\" (UniqueName: \"kubernetes.io/projected/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kube-api-access-5jrs8\") pod \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") "
Oct 30 22:23:08.247227 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.247163    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kserve-provision-location\") pod \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") "
Oct 30 22:23:08.247227 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.247186    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-proxy-tls\" (UniqueName: \"kubernetes.io/secret/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-proxy-tls\") pod \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") "
Oct 30 22:23:08.247227 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.247204    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kube-rbac-proxy-sar-config\") pod \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") "
Oct 30 22:23:08.247335 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.247230    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-cabundle-cert\") pod \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\" (UID: \"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe\") "
Oct 30 22:23:08.247568 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.247543    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kserve-provision-location" pod "d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" (UID: "d13da6d9-72cd-4c78-893f-6db0a8b1b9fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:08.247628 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.247572    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kube-rbac-proxy-sar-config" pod "d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" (UID: "d13da6d9-72cd-4c78-893f-6db0a8b1b9fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:08.247628 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.247618    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-cabundle-cert" pod "d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" (UID: "d13da6d9-72cd-4c78-893f-6db0a8b1b9fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:08.249849 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.249810    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-proxy-tls" pod "d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" (UID: "d13da6d9-72cd-4c78-893f-6db0a8b1b9fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:08.249993 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.249908    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kube-api-access-5jrs8" pod "d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" (UID: "d13da6d9-72cd-4c78-893f-6db0a8b1b9fe"). InnerVolumeSpecName "kube-api-access-5jrs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:08.348023 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.347992    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:08.348023 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.348021    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:08.348218 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.348034    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:08.348218 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.348047    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:08.348218 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.348058    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jrs8\" (UniqueName: \"kubernetes.io/projected/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe-kube-api-access-5jrs8\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:08.369345 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.369311    4473 generic.go:356] "Generic (PLEG): container finished" podID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" containerID="98842152522d0612fafe0d9d450b8e7252506240bca7473b1ebaae82ecbd1ab1" exitCode=0
Oct 30 22:23:08.369525 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.369365    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"
Oct 30 22:23:08.369525 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.369390    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv" event={"ID":"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe","Type":"ContainerDied","Data":"98842152522d0612fafe0d9d450b8e7252506240bca7473b1ebaae82ecbd1ab1"}
Oct 30 22:23:08.369525 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.369455    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv" event={"ID":"d13da6d9-72cd-4c78-893f-6db0a8b1b9fe","Type":"ContainerDied","Data":"e3adcb7fb2541850396c186a5b8e2a41e230c9a7e9166979ffda742fe28b26a9"}
Oct 30 22:23:08.369525 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.369493    4473 scope.go:122] "RemoveContainer" containerID="aaadfc88902b171c4d8f10f2e2a74e75995464443dec03be1fe8323a3fd9dd99"
Oct 30 22:23:08.373264 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.373235    4473 generic.go:356] "Generic (PLEG): container finished" podID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerID="4dbde38f4fe3e87950daf48e7742cf078482b84fd1069d0bc3bba4994498a9a2" exitCode=0
Oct 30 22:23:08.373400 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.373277    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" event={"ID":"dd5f7e4e-ead8-4b99-b829-b1bde61f8868","Type":"ContainerDied","Data":"4dbde38f4fe3e87950daf48e7742cf078482b84fd1069d0bc3bba4994498a9a2"}
Oct 30 22:23:08.379134 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.379089    4473 scope.go:122] "RemoveContainer" containerID="98842152522d0612fafe0d9d450b8e7252506240bca7473b1ebaae82ecbd1ab1"
Oct 30 22:23:08.386982 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.386937    4473 scope.go:122] "RemoveContainer" containerID="803cdf9344bbf90504157167f941e59b30dbe761e58160fe272dc167f5ab8ab2"
Oct 30 22:23:08.394203 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.394185    4473 scope.go:122] "RemoveContainer" containerID="aaadfc88902b171c4d8f10f2e2a74e75995464443dec03be1fe8323a3fd9dd99"
Oct 30 22:23:08.394458 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:23:08.394437    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaadfc88902b171c4d8f10f2e2a74e75995464443dec03be1fe8323a3fd9dd99\": container with ID starting with aaadfc88902b171c4d8f10f2e2a74e75995464443dec03be1fe8323a3fd9dd99 not found: ID does not exist" containerID="aaadfc88902b171c4d8f10f2e2a74e75995464443dec03be1fe8323a3fd9dd99"
Oct 30 22:23:08.394508 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.394470    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaadfc88902b171c4d8f10f2e2a74e75995464443dec03be1fe8323a3fd9dd99"} err="failed to get container status \"aaadfc88902b171c4d8f10f2e2a74e75995464443dec03be1fe8323a3fd9dd99\": rpc error: code = NotFound desc = could not find container \"aaadfc88902b171c4d8f10f2e2a74e75995464443dec03be1fe8323a3fd9dd99\": container with ID starting with aaadfc88902b171c4d8f10f2e2a74e75995464443dec03be1fe8323a3fd9dd99 not found: ID does not exist"
Oct 30 22:23:08.394508 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.394492    4473 scope.go:122] "RemoveContainer" containerID="98842152522d0612fafe0d9d450b8e7252506240bca7473b1ebaae82ecbd1ab1"
Oct 30 22:23:08.394745 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:23:08.394724    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98842152522d0612fafe0d9d450b8e7252506240bca7473b1ebaae82ecbd1ab1\": container with ID starting with 98842152522d0612fafe0d9d450b8e7252506240bca7473b1ebaae82ecbd1ab1 not found: ID does not exist" containerID="98842152522d0612fafe0d9d450b8e7252506240bca7473b1ebaae82ecbd1ab1"
Oct 30 22:23:08.394788 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.394747    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98842152522d0612fafe0d9d450b8e7252506240bca7473b1ebaae82ecbd1ab1"} err="failed to get container status \"98842152522d0612fafe0d9d450b8e7252506240bca7473b1ebaae82ecbd1ab1\": rpc error: code = NotFound desc = could not find container \"98842152522d0612fafe0d9d450b8e7252506240bca7473b1ebaae82ecbd1ab1\": container with ID starting with 98842152522d0612fafe0d9d450b8e7252506240bca7473b1ebaae82ecbd1ab1 not found: ID does not exist"
Oct 30 22:23:08.394788 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.394762    4473 scope.go:122] "RemoveContainer" containerID="803cdf9344bbf90504157167f941e59b30dbe761e58160fe272dc167f5ab8ab2"
Oct 30 22:23:08.395317 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:23:08.395301    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"803cdf9344bbf90504157167f941e59b30dbe761e58160fe272dc167f5ab8ab2\": container with ID starting with 803cdf9344bbf90504157167f941e59b30dbe761e58160fe272dc167f5ab8ab2 not found: ID does not exist" containerID="803cdf9344bbf90504157167f941e59b30dbe761e58160fe272dc167f5ab8ab2"
Oct 30 22:23:08.395359 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.395323    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803cdf9344bbf90504157167f941e59b30dbe761e58160fe272dc167f5ab8ab2"} err="failed to get container status \"803cdf9344bbf90504157167f941e59b30dbe761e58160fe272dc167f5ab8ab2\": rpc error: code = NotFound desc = could not find container \"803cdf9344bbf90504157167f941e59b30dbe761e58160fe272dc167f5ab8ab2\": container with ID starting with 803cdf9344bbf90504157167f941e59b30dbe761e58160fe272dc167f5ab8ab2 not found: ID does not exist"
Oct 30 22:23:08.540170 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.540132    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"]
Oct 30 22:23:08.548602 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.548576    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/sklearn-v2-mlserver-predictor-6569c8f4df-khkqv"]
Oct 30 22:23:08.899334 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.899294    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Oct 30 22:23:08.904761 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.904727    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 22:23:08.905116 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:08.905092    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:23:09.886197 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:09.886140    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" path="/var/lib/kubelet/pods/d13da6d9-72cd-4c78-893f-6db0a8b1b9fe/volumes"
Oct 30 22:23:10.794851 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:10.794820    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:11.000807 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.000769    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:11.303256 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:23:11.303221    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:23:11.530041 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.530010    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"]
Oct 30 22:23:11.530341 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.530330    4473 cpu_manager.go:414] "containerMap: removing container" podUID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" containerName="kserve-container"
Oct 30 22:23:11.530403 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.530343    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" containerName="kserve-container"
Oct 30 22:23:11.530403 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.530365    4473 cpu_manager.go:414] "containerMap: removing container" podUID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" containerName="storage-initializer"
Oct 30 22:23:11.530403 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.530371    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" containerName="storage-initializer"
Oct 30 22:23:11.530403 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.530386    4473 cpu_manager.go:414] "containerMap: removing container" podUID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" containerName="kube-rbac-proxy"
Oct 30 22:23:11.530403 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.530390    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" containerName="kube-rbac-proxy"
Oct 30 22:23:11.530596 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.530475    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" containerName="kube-rbac-proxy"
Oct 30 22:23:11.530596 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.530486    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13da6d9-72cd-4c78-893f-6db0a8b1b9fe" containerName="kserve-container"
Oct 30 22:23:11.533654 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.533634    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:11.575384 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.575313    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlzwt\" (UniqueName: \"kubernetes.io/projected/f2ad0634-8dbc-43a7-b091-41ddd996d801-kube-api-access-tlzwt\") pod \"isvc-sklearn-runtime-predictor-75cf99d79f-llpfs\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") " pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:11.575384 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.575354    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2ad0634-8dbc-43a7-b091-41ddd996d801-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-75cf99d79f-llpfs\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") " pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:11.575384 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.575378    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2ad0634-8dbc-43a7-b091-41ddd996d801-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-75cf99d79f-llpfs\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") " pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:11.575589 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.575474    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2ad0634-8dbc-43a7-b091-41ddd996d801-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-75cf99d79f-llpfs\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") " pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:11.575589 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.575565    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2ad0634-8dbc-43a7-b091-41ddd996d801-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-75cf99d79f-llpfs\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") " pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:11.587898 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.587873    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:23:11.638082 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.638047    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Oct 30 22:23:11.638082 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.638080    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:23:11.638284 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.638109    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:23:11.638284 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.638141    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:23:11.638284 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.638142    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Oct 30 22:23:11.638284 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.638213    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:23:11.639860 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.639842    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"]
Oct 30 22:23:11.676685 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.676653    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2ad0634-8dbc-43a7-b091-41ddd996d801-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-75cf99d79f-llpfs\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") " pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:11.676852 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.676698    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlzwt\" (UniqueName: \"kubernetes.io/projected/f2ad0634-8dbc-43a7-b091-41ddd996d801-kube-api-access-tlzwt\") pod \"isvc-sklearn-runtime-predictor-75cf99d79f-llpfs\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") " pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:11.676852 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.676729    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2ad0634-8dbc-43a7-b091-41ddd996d801-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-75cf99d79f-llpfs\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") " pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:11.676852 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.676764    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2ad0634-8dbc-43a7-b091-41ddd996d801-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-75cf99d79f-llpfs\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") " pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:11.676852 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.676794    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2ad0634-8dbc-43a7-b091-41ddd996d801-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-75cf99d79f-llpfs\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") " pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:11.677288 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.677266    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2ad0634-8dbc-43a7-b091-41ddd996d801-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-75cf99d79f-llpfs\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") " pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:11.677529 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.677508    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2ad0634-8dbc-43a7-b091-41ddd996d801-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-75cf99d79f-llpfs\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") " pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:11.677594 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.677511    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2ad0634-8dbc-43a7-b091-41ddd996d801-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-75cf99d79f-llpfs\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") " pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:11.679724 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.679701    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2ad0634-8dbc-43a7-b091-41ddd996d801-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-75cf99d79f-llpfs\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") " pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:11.735432 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.735392    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlzwt\" (UniqueName: \"kubernetes.io/projected/f2ad0634-8dbc-43a7-b091-41ddd996d801-kube-api-access-tlzwt\") pod \"isvc-sklearn-runtime-predictor-75cf99d79f-llpfs\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") " pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:11.846880 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:11.846793    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:12.273756 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:12.273722    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"]
Oct 30 22:23:12.276977 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:23:12.276934    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ad0634_8dbc_43a7_b091_41ddd996d801.slice/crio-d3ad122b19dcd2c27d92aefb2051a0502f22032bf9f50a69ea81340d6dca106a WatchSource:0}: Error finding container d3ad122b19dcd2c27d92aefb2051a0502f22032bf9f50a69ea81340d6dca106a: Status 404 returned error can't find the container with id d3ad122b19dcd2c27d92aefb2051a0502f22032bf9f50a69ea81340d6dca106a
Oct 30 22:23:12.399024 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:12.398987    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs" event={"ID":"f2ad0634-8dbc-43a7-b091-41ddd996d801","Type":"ContainerStarted","Data":"6713da41c37094f8b70864aaf7ff9d0fdebac83ebf774ead565cb806149a2c04"}
Oct 30 22:23:12.399188 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:12.399055    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs" event={"ID":"f2ad0634-8dbc-43a7-b091-41ddd996d801","Type":"ContainerStarted","Data":"d3ad122b19dcd2c27d92aefb2051a0502f22032bf9f50a69ea81340d6dca106a"}
Oct 30 22:23:13.899687 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:13.899653    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Oct 30 22:23:13.900082 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:13.899777    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:23:18.899465 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:18.899419    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Oct 30 22:23:18.905392 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:18.905356    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 22:23:18.905792 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:18.905765    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:23:19.429809 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:19.429773    4473 generic.go:356] "Generic (PLEG): container finished" podID="f2ad0634-8dbc-43a7-b091-41ddd996d801" containerID="6713da41c37094f8b70864aaf7ff9d0fdebac83ebf774ead565cb806149a2c04" exitCode=0
Oct 30 22:23:19.429986 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:19.429844    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs" event={"ID":"f2ad0634-8dbc-43a7-b091-41ddd996d801","Type":"ContainerDied","Data":"6713da41c37094f8b70864aaf7ff9d0fdebac83ebf774ead565cb806149a2c04"}
Oct 30 22:23:20.436597 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:20.436558    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs" event={"ID":"f2ad0634-8dbc-43a7-b091-41ddd996d801","Type":"ContainerStarted","Data":"f159e57e86306ec5979243e144de4fd3369920955c06e00d7c0ec1c23e5b04f1"}
Oct 30 22:23:20.437131 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:20.436623    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs" event={"ID":"f2ad0634-8dbc-43a7-b091-41ddd996d801","Type":"ContainerStarted","Data":"0a6d7e59201c1cd5484832f211d113910c776e76a26fc67beab02b1328b7f376"}
Oct 30 22:23:20.437131 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:20.436914    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:20.608228 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:20.608174    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs" podStartSLOduration=2.932640698 podStartE2EDuration="9.608158937s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.675518239s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:20.606401471 +0000 UTC m=+1818.842610820" watchObservedRunningTime="2026-10-30 22:23:20.608158937 +0000 UTC m=+1818.844368281"
Oct 30 22:23:21.440823 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:21.440789    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:21.442021 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:21.441996    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs" podUID="f2ad0634-8dbc-43a7-b091-41ddd996d801" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 22:23:22.444753 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:22.444709    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs" podUID="f2ad0634-8dbc-43a7-b091-41ddd996d801" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 22:23:23.899588 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:23.899549    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Oct 30 22:23:27.449393 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:27.449365    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:27.449935 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:27.449909    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs" podUID="f2ad0634-8dbc-43a7-b091-41ddd996d801" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 22:23:28.899686 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:28.899640    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Oct 30 22:23:28.905154 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:28.905113    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 22:23:28.905312 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:28.905248    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:23:28.905735 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:28.905701    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:23:28.905852 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:28.905841    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:23:32.811368 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:32.811341    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:23:32.962133 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:32.962096    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-proxy-tls\") pod \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") "
Oct 30 22:23:32.962339 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:32.962172    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kube-api-access-r2v5b\" (UniqueName: \"kubernetes.io/projected/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kube-api-access-r2v5b\") pod \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") "
Oct 30 22:23:32.962339 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:32.962232    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kube-rbac-proxy-sar-config\") pod \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") "
Oct 30 22:23:32.962473 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:32.962342    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-cabundle-cert\") pod \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") "
Oct 30 22:23:32.962473 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:32.962397    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kserve-provision-location\") pod \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\" (UID: \"dd5f7e4e-ead8-4b99-b829-b1bde61f8868\") "
Oct 30 22:23:32.962611 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:32.962587    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kube-rbac-proxy-sar-config" pod "dd5f7e4e-ead8-4b99-b829-b1bde61f8868" (UID: "dd5f7e4e-ead8-4b99-b829-b1bde61f8868"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:32.962676 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:32.962645    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-cabundle-cert" pod "dd5f7e4e-ead8-4b99-b829-b1bde61f8868" (UID: "dd5f7e4e-ead8-4b99-b829-b1bde61f8868"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:32.962746 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:32.962708    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kserve-provision-location" pod "dd5f7e4e-ead8-4b99-b829-b1bde61f8868" (UID: "dd5f7e4e-ead8-4b99-b829-b1bde61f8868"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:32.964460 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:32.964437    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kube-api-access-r2v5b" pod "dd5f7e4e-ead8-4b99-b829-b1bde61f8868" (UID: "dd5f7e4e-ead8-4b99-b829-b1bde61f8868"). InnerVolumeSpecName "kube-api-access-r2v5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:32.964768 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:32.964745    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-proxy-tls" pod "dd5f7e4e-ead8-4b99-b829-b1bde61f8868" (UID: "dd5f7e4e-ead8-4b99-b829-b1bde61f8868"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:33.063090 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.063056    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:33.063090 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.063085    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:33.063090 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.063094    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:33.063315 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.063104    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:33.063315 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.063113    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2v5b\" (UniqueName: \"kubernetes.io/projected/dd5f7e4e-ead8-4b99-b829-b1bde61f8868-kube-api-access-r2v5b\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:33.492705 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.492665    4473 generic.go:356] "Generic (PLEG): container finished" podID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerID="201b24217875af62872779cdf40830c1e09dc23bd77b9a6f17a4631b1e90042e" exitCode=0
Oct 30 22:23:33.492887 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.492719    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"
Oct 30 22:23:33.492887 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.492710    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" event={"ID":"dd5f7e4e-ead8-4b99-b829-b1bde61f8868","Type":"ContainerDied","Data":"201b24217875af62872779cdf40830c1e09dc23bd77b9a6f17a4631b1e90042e"}
Oct 30 22:23:33.492887 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.492856    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk" event={"ID":"dd5f7e4e-ead8-4b99-b829-b1bde61f8868","Type":"ContainerDied","Data":"2014ba29df32a1d75ffde2ce7b92e9336265a7b91d8311ebd7084b53b8d84096"}
Oct 30 22:23:33.493029 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.492887    4473 scope.go:122] "RemoveContainer" containerID="201b24217875af62872779cdf40830c1e09dc23bd77b9a6f17a4631b1e90042e"
Oct 30 22:23:33.502153 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.502136    4473 scope.go:122] "RemoveContainer" containerID="2c2cc95ef0a6bcfef1b52ea719239672391c9ebcf220f66ed4f792c718b3878c"
Oct 30 22:23:33.512646 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.512623    4473 scope.go:122] "RemoveContainer" containerID="4dbde38f4fe3e87950daf48e7742cf078482b84fd1069d0bc3bba4994498a9a2"
Oct 30 22:23:33.521997 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.521972    4473 scope.go:122] "RemoveContainer" containerID="1faf4e10340541fd88b5b8c726502605938c921e325a90888dad1e9c6c662ba2"
Oct 30 22:23:33.531924 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.531900    4473 scope.go:122] "RemoveContainer" containerID="201b24217875af62872779cdf40830c1e09dc23bd77b9a6f17a4631b1e90042e"
Oct 30 22:23:33.532269 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:23:33.532249    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201b24217875af62872779cdf40830c1e09dc23bd77b9a6f17a4631b1e90042e\": container with ID starting with 201b24217875af62872779cdf40830c1e09dc23bd77b9a6f17a4631b1e90042e not found: ID does not exist" containerID="201b24217875af62872779cdf40830c1e09dc23bd77b9a6f17a4631b1e90042e"
Oct 30 22:23:33.532367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.532278    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201b24217875af62872779cdf40830c1e09dc23bd77b9a6f17a4631b1e90042e"} err="failed to get container status \"201b24217875af62872779cdf40830c1e09dc23bd77b9a6f17a4631b1e90042e\": rpc error: code = NotFound desc = could not find container \"201b24217875af62872779cdf40830c1e09dc23bd77b9a6f17a4631b1e90042e\": container with ID starting with 201b24217875af62872779cdf40830c1e09dc23bd77b9a6f17a4631b1e90042e not found: ID does not exist"
Oct 30 22:23:33.532367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.532296    4473 scope.go:122] "RemoveContainer" containerID="2c2cc95ef0a6bcfef1b52ea719239672391c9ebcf220f66ed4f792c718b3878c"
Oct 30 22:23:33.532527 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:23:33.532511    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c2cc95ef0a6bcfef1b52ea719239672391c9ebcf220f66ed4f792c718b3878c\": container with ID starting with 2c2cc95ef0a6bcfef1b52ea719239672391c9ebcf220f66ed4f792c718b3878c not found: ID does not exist" containerID="2c2cc95ef0a6bcfef1b52ea719239672391c9ebcf220f66ed4f792c718b3878c"
Oct 30 22:23:33.532596 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.532530    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2cc95ef0a6bcfef1b52ea719239672391c9ebcf220f66ed4f792c718b3878c"} err="failed to get container status \"2c2cc95ef0a6bcfef1b52ea719239672391c9ebcf220f66ed4f792c718b3878c\": rpc error: code = NotFound desc = could not find container \"2c2cc95ef0a6bcfef1b52ea719239672391c9ebcf220f66ed4f792c718b3878c\": container with ID starting with 2c2cc95ef0a6bcfef1b52ea719239672391c9ebcf220f66ed4f792c718b3878c not found: ID does not exist"
Oct 30 22:23:33.532596 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.532553    4473 scope.go:122] "RemoveContainer" containerID="4dbde38f4fe3e87950daf48e7742cf078482b84fd1069d0bc3bba4994498a9a2"
Oct 30 22:23:33.532783 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:23:33.532747    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dbde38f4fe3e87950daf48e7742cf078482b84fd1069d0bc3bba4994498a9a2\": container with ID starting with 4dbde38f4fe3e87950daf48e7742cf078482b84fd1069d0bc3bba4994498a9a2 not found: ID does not exist" containerID="4dbde38f4fe3e87950daf48e7742cf078482b84fd1069d0bc3bba4994498a9a2"
Oct 30 22:23:33.532783 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.532762    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbde38f4fe3e87950daf48e7742cf078482b84fd1069d0bc3bba4994498a9a2"} err="failed to get container status \"4dbde38f4fe3e87950daf48e7742cf078482b84fd1069d0bc3bba4994498a9a2\": rpc error: code = NotFound desc = could not find container \"4dbde38f4fe3e87950daf48e7742cf078482b84fd1069d0bc3bba4994498a9a2\": container with ID starting with 4dbde38f4fe3e87950daf48e7742cf078482b84fd1069d0bc3bba4994498a9a2 not found: ID does not exist"
Oct 30 22:23:33.532783 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.532773    4473 scope.go:122] "RemoveContainer" containerID="1faf4e10340541fd88b5b8c726502605938c921e325a90888dad1e9c6c662ba2"
Oct 30 22:23:33.533458 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:23:33.533018    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1faf4e10340541fd88b5b8c726502605938c921e325a90888dad1e9c6c662ba2\": container with ID starting with 1faf4e10340541fd88b5b8c726502605938c921e325a90888dad1e9c6c662ba2 not found: ID does not exist" containerID="1faf4e10340541fd88b5b8c726502605938c921e325a90888dad1e9c6c662ba2"
Oct 30 22:23:33.533458 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.533033    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1faf4e10340541fd88b5b8c726502605938c921e325a90888dad1e9c6c662ba2"} err="failed to get container status \"1faf4e10340541fd88b5b8c726502605938c921e325a90888dad1e9c6c662ba2\": rpc error: code = NotFound desc = could not find container \"1faf4e10340541fd88b5b8c726502605938c921e325a90888dad1e9c6c662ba2\": container with ID starting with 1faf4e10340541fd88b5b8c726502605938c921e325a90888dad1e9c6c662ba2 not found: ID does not exist"
Oct 30 22:23:33.710017 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.709985    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"]
Oct 30 22:23:33.760720 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.760636    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-logger-predictor-746864dcdc-9wnjk"]
Oct 30 22:23:33.887270 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:33.887234    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" path="/var/lib/kubelet/pods/dd5f7e4e-ead8-4b99-b829-b1bde61f8868/volumes"
Oct 30 22:23:35.680734 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:35.680688    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:35.837795 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:35.837753    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:37.450551 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:37.450525    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:41.304310 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:23:41.304271    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:23:45.278206 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:45.278118    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"]
Oct 30 22:23:45.278658 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:45.278423    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs" podUID="f2ad0634-8dbc-43a7-b091-41ddd996d801" containerName="kserve-container" containerID="cri-o://0a6d7e59201c1cd5484832f211d113910c776e76a26fc67beab02b1328b7f376" gracePeriod=30
Oct 30 22:23:45.278658 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:45.278447    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs" podUID="f2ad0634-8dbc-43a7-b091-41ddd996d801" containerName="kube-rbac-proxy" containerID="cri-o://f159e57e86306ec5979243e144de4fd3369920955c06e00d7c0ec1c23e5b04f1" gracePeriod=30
Oct 30 22:23:45.282849 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:45.282813    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:45.549586 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:45.549500    4473 generic.go:356] "Generic (PLEG): container finished" podID="f2ad0634-8dbc-43a7-b091-41ddd996d801" containerID="f159e57e86306ec5979243e144de4fd3369920955c06e00d7c0ec1c23e5b04f1" exitCode=0
Oct 30 22:23:45.549586 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:45.549567    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs" event={"ID":"f2ad0634-8dbc-43a7-b091-41ddd996d801","Type":"ContainerDied","Data":"f159e57e86306ec5979243e144de4fd3369920955c06e00d7c0ec1c23e5b04f1"}
Oct 30 22:23:46.122136 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.122110    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:46.280627 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.280541    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ad0634-8dbc-43a7-b091-41ddd996d801-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2ad0634-8dbc-43a7-b091-41ddd996d801-cabundle-cert\") pod \"f2ad0634-8dbc-43a7-b091-41ddd996d801\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") "
Oct 30 22:23:46.280627 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.280578    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ad0634-8dbc-43a7-b091-41ddd996d801-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2ad0634-8dbc-43a7-b091-41ddd996d801-kube-rbac-proxy-sar-config\") pod \"f2ad0634-8dbc-43a7-b091-41ddd996d801\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") "
Oct 30 22:23:46.280627 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.280628    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ad0634-8dbc-43a7-b091-41ddd996d801-kube-api-access-tlzwt\" (UniqueName: \"kubernetes.io/projected/f2ad0634-8dbc-43a7-b091-41ddd996d801-kube-api-access-tlzwt\") pod \"f2ad0634-8dbc-43a7-b091-41ddd996d801\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") "
Oct 30 22:23:46.281251 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.280651    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ad0634-8dbc-43a7-b091-41ddd996d801-proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2ad0634-8dbc-43a7-b091-41ddd996d801-proxy-tls\") pod \"f2ad0634-8dbc-43a7-b091-41ddd996d801\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") "
Oct 30 22:23:46.281251 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.280703    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ad0634-8dbc-43a7-b091-41ddd996d801-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2ad0634-8dbc-43a7-b091-41ddd996d801-kserve-provision-location\") pod \"f2ad0634-8dbc-43a7-b091-41ddd996d801\" (UID: \"f2ad0634-8dbc-43a7-b091-41ddd996d801\") "
Oct 30 22:23:46.281251 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.281051    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ad0634-8dbc-43a7-b091-41ddd996d801-cabundle-cert" pod "f2ad0634-8dbc-43a7-b091-41ddd996d801" (UID: "f2ad0634-8dbc-43a7-b091-41ddd996d801"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:46.281251 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.281115    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ad0634-8dbc-43a7-b091-41ddd996d801-kube-rbac-proxy-sar-config" pod "f2ad0634-8dbc-43a7-b091-41ddd996d801" (UID: "f2ad0634-8dbc-43a7-b091-41ddd996d801"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:46.283078 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.283050    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ad0634-8dbc-43a7-b091-41ddd996d801-proxy-tls" pod "f2ad0634-8dbc-43a7-b091-41ddd996d801" (UID: "f2ad0634-8dbc-43a7-b091-41ddd996d801"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:46.283078 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.283070    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ad0634-8dbc-43a7-b091-41ddd996d801-kube-api-access-tlzwt" pod "f2ad0634-8dbc-43a7-b091-41ddd996d801" (UID: "f2ad0634-8dbc-43a7-b091-41ddd996d801"). InnerVolumeSpecName "kube-api-access-tlzwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:46.309375 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.309336    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ad0634-8dbc-43a7-b091-41ddd996d801-kserve-provision-location" pod "f2ad0634-8dbc-43a7-b091-41ddd996d801" (UID: "f2ad0634-8dbc-43a7-b091-41ddd996d801"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:46.381414 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.381378    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2ad0634-8dbc-43a7-b091-41ddd996d801-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:46.381414 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.381408    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2ad0634-8dbc-43a7-b091-41ddd996d801-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:46.381414 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.381417    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2ad0634-8dbc-43a7-b091-41ddd996d801-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:46.381657 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.381427    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlzwt\" (UniqueName: \"kubernetes.io/projected/f2ad0634-8dbc-43a7-b091-41ddd996d801-kube-api-access-tlzwt\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:46.381657 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.381436    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2ad0634-8dbc-43a7-b091-41ddd996d801-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:46.555586 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.555497    4473 generic.go:356] "Generic (PLEG): container finished" podID="f2ad0634-8dbc-43a7-b091-41ddd996d801" containerID="0a6d7e59201c1cd5484832f211d113910c776e76a26fc67beab02b1328b7f376" exitCode=0
Oct 30 22:23:46.555586 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.555554    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"
Oct 30 22:23:46.555779 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.555590    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs" event={"ID":"f2ad0634-8dbc-43a7-b091-41ddd996d801","Type":"ContainerDied","Data":"0a6d7e59201c1cd5484832f211d113910c776e76a26fc67beab02b1328b7f376"}
Oct 30 22:23:46.555779 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.555654    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs" event={"ID":"f2ad0634-8dbc-43a7-b091-41ddd996d801","Type":"ContainerDied","Data":"d3ad122b19dcd2c27d92aefb2051a0502f22032bf9f50a69ea81340d6dca106a"}
Oct 30 22:23:46.555779 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.555683    4473 scope.go:122] "RemoveContainer" containerID="f159e57e86306ec5979243e144de4fd3369920955c06e00d7c0ec1c23e5b04f1"
Oct 30 22:23:46.564335 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.564320    4473 scope.go:122] "RemoveContainer" containerID="0a6d7e59201c1cd5484832f211d113910c776e76a26fc67beab02b1328b7f376"
Oct 30 22:23:46.571768 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.571750    4473 scope.go:122] "RemoveContainer" containerID="6713da41c37094f8b70864aaf7ff9d0fdebac83ebf774ead565cb806149a2c04"
Oct 30 22:23:46.578921 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.578901    4473 scope.go:122] "RemoveContainer" containerID="f159e57e86306ec5979243e144de4fd3369920955c06e00d7c0ec1c23e5b04f1"
Oct 30 22:23:46.579184 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:23:46.579164    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f159e57e86306ec5979243e144de4fd3369920955c06e00d7c0ec1c23e5b04f1\": container with ID starting with f159e57e86306ec5979243e144de4fd3369920955c06e00d7c0ec1c23e5b04f1 not found: ID does not exist" containerID="f159e57e86306ec5979243e144de4fd3369920955c06e00d7c0ec1c23e5b04f1"
Oct 30 22:23:46.579258 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.579197    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f159e57e86306ec5979243e144de4fd3369920955c06e00d7c0ec1c23e5b04f1"} err="failed to get container status \"f159e57e86306ec5979243e144de4fd3369920955c06e00d7c0ec1c23e5b04f1\": rpc error: code = NotFound desc = could not find container \"f159e57e86306ec5979243e144de4fd3369920955c06e00d7c0ec1c23e5b04f1\": container with ID starting with f159e57e86306ec5979243e144de4fd3369920955c06e00d7c0ec1c23e5b04f1 not found: ID does not exist"
Oct 30 22:23:46.579258 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.579221    4473 scope.go:122] "RemoveContainer" containerID="0a6d7e59201c1cd5484832f211d113910c776e76a26fc67beab02b1328b7f376"
Oct 30 22:23:46.579498 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:23:46.579481    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6d7e59201c1cd5484832f211d113910c776e76a26fc67beab02b1328b7f376\": container with ID starting with 0a6d7e59201c1cd5484832f211d113910c776e76a26fc67beab02b1328b7f376 not found: ID does not exist" containerID="0a6d7e59201c1cd5484832f211d113910c776e76a26fc67beab02b1328b7f376"
Oct 30 22:23:46.579537 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.579513    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6d7e59201c1cd5484832f211d113910c776e76a26fc67beab02b1328b7f376"} err="failed to get container status \"0a6d7e59201c1cd5484832f211d113910c776e76a26fc67beab02b1328b7f376\": rpc error: code = NotFound desc = could not find container \"0a6d7e59201c1cd5484832f211d113910c776e76a26fc67beab02b1328b7f376\": container with ID starting with 0a6d7e59201c1cd5484832f211d113910c776e76a26fc67beab02b1328b7f376 not found: ID does not exist"
Oct 30 22:23:46.579537 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.579529    4473 scope.go:122] "RemoveContainer" containerID="6713da41c37094f8b70864aaf7ff9d0fdebac83ebf774ead565cb806149a2c04"
Oct 30 22:23:46.579750 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:23:46.579733    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6713da41c37094f8b70864aaf7ff9d0fdebac83ebf774ead565cb806149a2c04\": container with ID starting with 6713da41c37094f8b70864aaf7ff9d0fdebac83ebf774ead565cb806149a2c04 not found: ID does not exist" containerID="6713da41c37094f8b70864aaf7ff9d0fdebac83ebf774ead565cb806149a2c04"
Oct 30 22:23:46.579806 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.579757    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6713da41c37094f8b70864aaf7ff9d0fdebac83ebf774ead565cb806149a2c04"} err="failed to get container status \"6713da41c37094f8b70864aaf7ff9d0fdebac83ebf774ead565cb806149a2c04\": rpc error: code = NotFound desc = could not find container \"6713da41c37094f8b70864aaf7ff9d0fdebac83ebf774ead565cb806149a2c04\": container with ID starting with 6713da41c37094f8b70864aaf7ff9d0fdebac83ebf774ead565cb806149a2c04 not found: ID does not exist"
Oct 30 22:23:46.721930 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.721900    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"]
Oct 30 22:23:46.772421 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:46.772391    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-runtime-predictor-75cf99d79f-llpfs"]
Oct 30 22:23:47.885678 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:47.885643    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ad0634-8dbc-43a7-b091-41ddd996d801" path="/var/lib/kubelet/pods/f2ad0634-8dbc-43a7-b091-41ddd996d801/volumes"
Oct 30 22:23:49.714975 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:49.714916    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:49.864591 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:49.864554    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:50.206847 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.206814    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"]
Oct 30 22:23:50.207144 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207130    4473 cpu_manager.go:414] "containerMap: removing container" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kube-rbac-proxy"
Oct 30 22:23:50.207193 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207145    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kube-rbac-proxy"
Oct 30 22:23:50.207193 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207165    4473 cpu_manager.go:414] "containerMap: removing container" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="agent"
Oct 30 22:23:50.207193 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207170    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="agent"
Oct 30 22:23:50.207193 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207182    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f2ad0634-8dbc-43a7-b091-41ddd996d801" containerName="kserve-container"
Oct 30 22:23:50.207193 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207187    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ad0634-8dbc-43a7-b091-41ddd996d801" containerName="kserve-container"
Oct 30 22:23:50.207344 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207196    4473 cpu_manager.go:414] "containerMap: removing container" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kserve-container"
Oct 30 22:23:50.207344 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207202    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kserve-container"
Oct 30 22:23:50.207344 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207211    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f2ad0634-8dbc-43a7-b091-41ddd996d801" containerName="kube-rbac-proxy"
Oct 30 22:23:50.207344 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207215    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ad0634-8dbc-43a7-b091-41ddd996d801" containerName="kube-rbac-proxy"
Oct 30 22:23:50.207344 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207221    4473 cpu_manager.go:414] "containerMap: removing container" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="storage-initializer"
Oct 30 22:23:50.207344 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207225    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="storage-initializer"
Oct 30 22:23:50.207344 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207231    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f2ad0634-8dbc-43a7-b091-41ddd996d801" containerName="storage-initializer"
Oct 30 22:23:50.207344 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207235    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ad0634-8dbc-43a7-b091-41ddd996d801" containerName="storage-initializer"
Oct 30 22:23:50.207344 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207280    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ad0634-8dbc-43a7-b091-41ddd996d801" containerName="kserve-container"
Oct 30 22:23:50.207344 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207288    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kserve-container"
Oct 30 22:23:50.207344 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207294    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ad0634-8dbc-43a7-b091-41ddd996d801" containerName="kube-rbac-proxy"
Oct 30 22:23:50.207344 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207300    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="kube-rbac-proxy"
Oct 30 22:23:50.207344 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.207309    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd5f7e4e-ead8-4b99-b829-b1bde61f8868" containerName="agent"
Oct 30 22:23:50.211894 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.211878    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:50.265172 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.265145    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:23:50.265172 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.265145    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:23:50.265362 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.265145    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 22:23:50.265362 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.265212    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:23:50.310055 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.310026    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7688668-874d-45cc-86ab-575d75057972-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:50.310196 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.310080    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87n7r\" (UniqueName: \"kubernetes.io/projected/b7688668-874d-45cc-86ab-575d75057972-kube-api-access-87n7r\") pod \"isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:50.310196 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.310162    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7688668-874d-45cc-86ab-575d75057972-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:50.310262 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.310198    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7688668-874d-45cc-86ab-575d75057972-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:50.310292 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.310262    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7688668-874d-45cc-86ab-575d75057972-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:50.315589 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.315565    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:23:50.315730 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.315599    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Oct 30 22:23:50.315730 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.315662    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:23:50.317189 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.317167    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"]
Oct 30 22:23:50.410613 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.410579    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7688668-874d-45cc-86ab-575d75057972-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:50.410613 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.410615    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7688668-874d-45cc-86ab-575d75057972-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:50.410872 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.410659    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7688668-874d-45cc-86ab-575d75057972-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:50.410872 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.410690    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7688668-874d-45cc-86ab-575d75057972-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:50.410872 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.410725    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87n7r\" (UniqueName: \"kubernetes.io/projected/b7688668-874d-45cc-86ab-575d75057972-kube-api-access-87n7r\") pod \"isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:50.411209 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.411186    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7688668-874d-45cc-86ab-575d75057972-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:50.411517 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.411500    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7688668-874d-45cc-86ab-575d75057972-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:50.411559 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.411501    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7688668-874d-45cc-86ab-575d75057972-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:50.413165 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.413150    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7688668-874d-45cc-86ab-575d75057972-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:50.468277 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.468202    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87n7r\" (UniqueName: \"kubernetes.io/projected/b7688668-874d-45cc-86ab-575d75057972-kube-api-access-87n7r\") pod \"isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:50.528937 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.528902    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:50.954458 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:50.954426    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"]
Oct 30 22:23:50.956942 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:23:50.956913    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7688668_874d_45cc_86ab_575d75057972.slice/crio-98495c625897a1bd279f5e08677ce86a3dab57b5afee05430834dd0d2fa8a9d5 WatchSource:0}: Error finding container 98495c625897a1bd279f5e08677ce86a3dab57b5afee05430834dd0d2fa8a9d5: Status 404 returned error can't find the container with id 98495c625897a1bd279f5e08677ce86a3dab57b5afee05430834dd0d2fa8a9d5
Oct 30 22:23:51.580833 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:51.580791    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9" event={"ID":"b7688668-874d-45cc-86ab-575d75057972","Type":"ContainerStarted","Data":"a20a4ede9d91782947252ac8e134c4961556d3636e45d33b991920ee0ea395f5"}
Oct 30 22:23:51.581060 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:51.580851    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9" event={"ID":"b7688668-874d-45cc-86ab-575d75057972","Type":"ContainerStarted","Data":"98495c625897a1bd279f5e08677ce86a3dab57b5afee05430834dd0d2fa8a9d5"}
Oct 30 22:23:55.605353 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:55.605317    4473 generic.go:356] "Generic (PLEG): container finished" podID="b7688668-874d-45cc-86ab-575d75057972" containerID="a20a4ede9d91782947252ac8e134c4961556d3636e45d33b991920ee0ea395f5" exitCode=0
Oct 30 22:23:55.605883 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:55.605383    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9" event={"ID":"b7688668-874d-45cc-86ab-575d75057972","Type":"ContainerDied","Data":"a20a4ede9d91782947252ac8e134c4961556d3636e45d33b991920ee0ea395f5"}
Oct 30 22:23:56.611207 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:56.611173    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9" event={"ID":"b7688668-874d-45cc-86ab-575d75057972","Type":"ContainerStarted","Data":"9881da2d2e6aef37ec78ee76fafa3ef4f59a251c251a50d5d5f1153f065356b6"}
Oct 30 22:23:56.611596 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:56.611233    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9" event={"ID":"b7688668-874d-45cc-86ab-575d75057972","Type":"ContainerStarted","Data":"496672be8c81bed8b82b419a79f61bcd57c78498bb2eac76b207f43166c0ffcd"}
Oct 30 22:23:56.611596 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:56.611535    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:56.611668 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:56.611599    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:23:56.780906 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:23:56.780848    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9" podStartSLOduration=2.778271028 podStartE2EDuration="6.780832448s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.00256142s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:56.780691709 +0000 UTC m=+1855.016901055" watchObservedRunningTime="2026-10-30 22:23:56.780832448 +0000 UTC m=+1855.017041792"
Oct 30 22:24:01.773517 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:01.773479    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:02.620810 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:02.620783    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:24:11.303717 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:24:11.303676    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:24:32.665198 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:32.665151    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9" podUID="b7688668-874d-45cc-86ab-575d75057972" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 30 22:24:41.302931 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:24:41.302893    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:24:42.623226 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:42.623194    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:24:50.446324 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:50.446290    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"]
Oct 30 22:24:50.446776 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:50.446535    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9" podUID="b7688668-874d-45cc-86ab-575d75057972" containerName="kserve-container" containerID="cri-o://496672be8c81bed8b82b419a79f61bcd57c78498bb2eac76b207f43166c0ffcd" gracePeriod=30
Oct 30 22:24:50.446776 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:50.446617    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9" podUID="b7688668-874d-45cc-86ab-575d75057972" containerName="kube-rbac-proxy" containerID="cri-o://9881da2d2e6aef37ec78ee76fafa3ef4f59a251c251a50d5d5f1153f065356b6" gracePeriod=30
Oct 30 22:24:50.447427 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:50.447400    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:50.836650 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:50.836570    4473 generic.go:356] "Generic (PLEG): container finished" podID="b7688668-874d-45cc-86ab-575d75057972" containerID="9881da2d2e6aef37ec78ee76fafa3ef4f59a251c251a50d5d5f1153f065356b6" exitCode=0
Oct 30 22:24:50.836650 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:50.836612    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9" event={"ID":"b7688668-874d-45cc-86ab-575d75057972","Type":"ContainerDied","Data":"9881da2d2e6aef37ec78ee76fafa3ef4f59a251c251a50d5d5f1153f065356b6"}
Oct 30 22:24:52.615668 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:52.615631    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9" podUID="b7688668-874d-45cc-86ab-575d75057972" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Oct 30 22:24:53.662154 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:53.662108    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9" podUID="b7688668-874d-45cc-86ab-575d75057972" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.44:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:57.615552 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.615520    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9" podUID="b7688668-874d-45cc-86ab-575d75057972" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Oct 30 22:24:57.800139 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.800115    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:24:57.866783 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.866708    4473 generic.go:356] "Generic (PLEG): container finished" podID="b7688668-874d-45cc-86ab-575d75057972" containerID="496672be8c81bed8b82b419a79f61bcd57c78498bb2eac76b207f43166c0ffcd" exitCode=0
Oct 30 22:24:57.866917 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.866788    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"
Oct 30 22:24:57.866917 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.866786    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9" event={"ID":"b7688668-874d-45cc-86ab-575d75057972","Type":"ContainerDied","Data":"496672be8c81bed8b82b419a79f61bcd57c78498bb2eac76b207f43166c0ffcd"}
Oct 30 22:24:57.867021 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.866918    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9" event={"ID":"b7688668-874d-45cc-86ab-575d75057972","Type":"ContainerDied","Data":"98495c625897a1bd279f5e08677ce86a3dab57b5afee05430834dd0d2fa8a9d5"}
Oct 30 22:24:57.867021 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.866945    4473 scope.go:122] "RemoveContainer" containerID="9881da2d2e6aef37ec78ee76fafa3ef4f59a251c251a50d5d5f1153f065356b6"
Oct 30 22:24:57.877899 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.877881    4473 scope.go:122] "RemoveContainer" containerID="496672be8c81bed8b82b419a79f61bcd57c78498bb2eac76b207f43166c0ffcd"
Oct 30 22:24:57.886138 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.886120    4473 scope.go:122] "RemoveContainer" containerID="a20a4ede9d91782947252ac8e134c4961556d3636e45d33b991920ee0ea395f5"
Oct 30 22:24:57.893334 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.893314    4473 scope.go:122] "RemoveContainer" containerID="9881da2d2e6aef37ec78ee76fafa3ef4f59a251c251a50d5d5f1153f065356b6"
Oct 30 22:24:57.893594 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:24:57.893577    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9881da2d2e6aef37ec78ee76fafa3ef4f59a251c251a50d5d5f1153f065356b6\": container with ID starting with 9881da2d2e6aef37ec78ee76fafa3ef4f59a251c251a50d5d5f1153f065356b6 not found: ID does not exist" containerID="9881da2d2e6aef37ec78ee76fafa3ef4f59a251c251a50d5d5f1153f065356b6"
Oct 30 22:24:57.893656 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.893602    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9881da2d2e6aef37ec78ee76fafa3ef4f59a251c251a50d5d5f1153f065356b6"} err="failed to get container status \"9881da2d2e6aef37ec78ee76fafa3ef4f59a251c251a50d5d5f1153f065356b6\": rpc error: code = NotFound desc = could not find container \"9881da2d2e6aef37ec78ee76fafa3ef4f59a251c251a50d5d5f1153f065356b6\": container with ID starting with 9881da2d2e6aef37ec78ee76fafa3ef4f59a251c251a50d5d5f1153f065356b6 not found: ID does not exist"
Oct 30 22:24:57.893656 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.893618    4473 scope.go:122] "RemoveContainer" containerID="496672be8c81bed8b82b419a79f61bcd57c78498bb2eac76b207f43166c0ffcd"
Oct 30 22:24:57.893881 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:24:57.893863    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"496672be8c81bed8b82b419a79f61bcd57c78498bb2eac76b207f43166c0ffcd\": container with ID starting with 496672be8c81bed8b82b419a79f61bcd57c78498bb2eac76b207f43166c0ffcd not found: ID does not exist" containerID="496672be8c81bed8b82b419a79f61bcd57c78498bb2eac76b207f43166c0ffcd"
Oct 30 22:24:57.893932 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.893887    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"496672be8c81bed8b82b419a79f61bcd57c78498bb2eac76b207f43166c0ffcd"} err="failed to get container status \"496672be8c81bed8b82b419a79f61bcd57c78498bb2eac76b207f43166c0ffcd\": rpc error: code = NotFound desc = could not find container \"496672be8c81bed8b82b419a79f61bcd57c78498bb2eac76b207f43166c0ffcd\": container with ID starting with 496672be8c81bed8b82b419a79f61bcd57c78498bb2eac76b207f43166c0ffcd not found: ID does not exist"
Oct 30 22:24:57.893932 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.893901    4473 scope.go:122] "RemoveContainer" containerID="a20a4ede9d91782947252ac8e134c4961556d3636e45d33b991920ee0ea395f5"
Oct 30 22:24:57.894143 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:24:57.894129    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a20a4ede9d91782947252ac8e134c4961556d3636e45d33b991920ee0ea395f5\": container with ID starting with a20a4ede9d91782947252ac8e134c4961556d3636e45d33b991920ee0ea395f5 not found: ID does not exist" containerID="a20a4ede9d91782947252ac8e134c4961556d3636e45d33b991920ee0ea395f5"
Oct 30 22:24:57.894189 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.894148    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20a4ede9d91782947252ac8e134c4961556d3636e45d33b991920ee0ea395f5"} err="failed to get container status \"a20a4ede9d91782947252ac8e134c4961556d3636e45d33b991920ee0ea395f5\": rpc error: code = NotFound desc = could not find container \"a20a4ede9d91782947252ac8e134c4961556d3636e45d33b991920ee0ea395f5\": container with ID starting with a20a4ede9d91782947252ac8e134c4961556d3636e45d33b991920ee0ea395f5 not found: ID does not exist"
Oct 30 22:24:57.909367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.909349    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7688668-874d-45cc-86ab-575d75057972-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7688668-874d-45cc-86ab-575d75057972-cabundle-cert\") pod \"b7688668-874d-45cc-86ab-575d75057972\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") "
Oct 30 22:24:57.909437 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.909392    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7688668-874d-45cc-86ab-575d75057972-kube-api-access-87n7r\" (UniqueName: \"kubernetes.io/projected/b7688668-874d-45cc-86ab-575d75057972-kube-api-access-87n7r\") pod \"b7688668-874d-45cc-86ab-575d75057972\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") "
Oct 30 22:24:57.909437 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.909427    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7688668-874d-45cc-86ab-575d75057972-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7688668-874d-45cc-86ab-575d75057972-kserve-provision-location\") pod \"b7688668-874d-45cc-86ab-575d75057972\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") "
Oct 30 22:24:57.909522 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.909474    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7688668-874d-45cc-86ab-575d75057972-proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7688668-874d-45cc-86ab-575d75057972-proxy-tls\") pod \"b7688668-874d-45cc-86ab-575d75057972\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") "
Oct 30 22:24:57.909522 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.909493    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7688668-874d-45cc-86ab-575d75057972-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7688668-874d-45cc-86ab-575d75057972-kube-rbac-proxy-sar-config\") pod \"b7688668-874d-45cc-86ab-575d75057972\" (UID: \"b7688668-874d-45cc-86ab-575d75057972\") "
Oct 30 22:24:57.909807 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.909782    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7688668-874d-45cc-86ab-575d75057972-cabundle-cert" pod "b7688668-874d-45cc-86ab-575d75057972" (UID: "b7688668-874d-45cc-86ab-575d75057972"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:57.909807 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.909796    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7688668-874d-45cc-86ab-575d75057972-kserve-provision-location" pod "b7688668-874d-45cc-86ab-575d75057972" (UID: "b7688668-874d-45cc-86ab-575d75057972"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:57.909906 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.909883    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7688668-874d-45cc-86ab-575d75057972-kube-rbac-proxy-sar-config" pod "b7688668-874d-45cc-86ab-575d75057972" (UID: "b7688668-874d-45cc-86ab-575d75057972"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:57.911495 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.911476    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7688668-874d-45cc-86ab-575d75057972-kube-api-access-87n7r" pod "b7688668-874d-45cc-86ab-575d75057972" (UID: "b7688668-874d-45cc-86ab-575d75057972"). InnerVolumeSpecName "kube-api-access-87n7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:57.911662 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:57.911642    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7688668-874d-45cc-86ab-575d75057972-proxy-tls" pod "b7688668-874d-45cc-86ab-575d75057972" (UID: "b7688668-874d-45cc-86ab-575d75057972"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:58.010676 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:58.010645    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7688668-874d-45cc-86ab-575d75057972-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.010676 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:58.010672    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87n7r\" (UniqueName: \"kubernetes.io/projected/b7688668-874d-45cc-86ab-575d75057972-kube-api-access-87n7r\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.010916 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:58.010683    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7688668-874d-45cc-86ab-575d75057972-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.010916 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:58.010694    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7688668-874d-45cc-86ab-575d75057972-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.010916 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:58.010706    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7688668-874d-45cc-86ab-575d75057972-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.287635 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:58.287601    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"]
Oct 30 22:24:58.291648 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:58.291618    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-runtime-predictor-84854c9949-2j2v9"]
Oct 30 22:24:59.885278 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:24:59.885246    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7688668-874d-45cc-86ab-575d75057972" path="/var/lib/kubelet/pods/b7688668-874d-45cc-86ab-575d75057972/volumes"
Oct 30 22:25:01.048079 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.048041    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:01.268019 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.267990    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:01.528206 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.528178    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"]
Oct 30 22:25:01.528484 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.528472    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b7688668-874d-45cc-86ab-575d75057972" containerName="kserve-container"
Oct 30 22:25:01.528539 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.528485    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7688668-874d-45cc-86ab-575d75057972" containerName="kserve-container"
Oct 30 22:25:01.528539 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.528507    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b7688668-874d-45cc-86ab-575d75057972" containerName="storage-initializer"
Oct 30 22:25:01.528539 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.528514    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7688668-874d-45cc-86ab-575d75057972" containerName="storage-initializer"
Oct 30 22:25:01.528539 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.528521    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b7688668-874d-45cc-86ab-575d75057972" containerName="kube-rbac-proxy"
Oct 30 22:25:01.528539 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.528525    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7688668-874d-45cc-86ab-575d75057972" containerName="kube-rbac-proxy"
Oct 30 22:25:01.528709 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.528586    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7688668-874d-45cc-86ab-575d75057972" containerName="kube-rbac-proxy"
Oct 30 22:25:01.528709 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.528598    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7688668-874d-45cc-86ab-575d75057972" containerName="kserve-container"
Oct 30 22:25:01.533416 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.533398    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:01.586769 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.586734    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:25:01.586769 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.586743    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:25:01.637308 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.637267    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Oct 30 22:25:01.637308 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.637302    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:25:01.637308 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.637273    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:25:01.637308 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.637267    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Oct 30 22:25:01.637689 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.637453    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:25:01.638794 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.638771    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"]
Oct 30 22:25:01.639079 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.639058    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7db6554897-vrnjt\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:01.639127 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.639096    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7db6554897-vrnjt\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:01.639127 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.639120    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7db6554897-vrnjt\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:01.639248 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.639236    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7db6554897-vrnjt\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:01.639285 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.639257    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q46v6\" (UniqueName: \"kubernetes.io/projected/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kube-api-access-q46v6\") pod \"isvc-sklearn-v2-predictor-7db6554897-vrnjt\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:01.740177 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.740141    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7db6554897-vrnjt\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:01.740177 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.740178    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46v6\" (UniqueName: \"kubernetes.io/projected/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kube-api-access-q46v6\") pod \"isvc-sklearn-v2-predictor-7db6554897-vrnjt\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:01.740407 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.740311    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7db6554897-vrnjt\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:01.740407 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.740363    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7db6554897-vrnjt\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:01.740407 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.740396    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7db6554897-vrnjt\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:01.740743 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.740720    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7db6554897-vrnjt\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:01.741010 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.740992    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7db6554897-vrnjt\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:01.741096 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.741083    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7db6554897-vrnjt\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:01.742775 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.742757    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7db6554897-vrnjt\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:01.801644 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.801574    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46v6\" (UniqueName: \"kubernetes.io/projected/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kube-api-access-q46v6\") pod \"isvc-sklearn-v2-predictor-7db6554897-vrnjt\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:01.902037 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.902010    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:25:01.910682 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:01.910663    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:02.073888 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:02.073803    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:02.335746 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:02.335717    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"]
Oct 30 22:25:02.337698 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:25:02.337673    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e46be5_ef5b_44a2_bd61_bdf6ca28f7c2.slice/crio-cd63ccfc41dab82deb21b6e2bfe16989acf398654f6cf24816c3da771fbe71c6 WatchSource:0}: Error finding container cd63ccfc41dab82deb21b6e2bfe16989acf398654f6cf24816c3da771fbe71c6: Status 404 returned error can't find the container with id cd63ccfc41dab82deb21b6e2bfe16989acf398654f6cf24816c3da771fbe71c6
Oct 30 22:25:02.391766 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:02.391734    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:25:02.890564 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:02.890524    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" event={"ID":"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2","Type":"ContainerStarted","Data":"3133aa3a8486f9bf7e6c2425a71c758a7191d5d6ffa581b7682ed6268f3d6528"}
Oct 30 22:25:02.890760 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:02.890588    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" event={"ID":"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2","Type":"ContainerStarted","Data":"cd63ccfc41dab82deb21b6e2bfe16989acf398654f6cf24816c3da771fbe71c6"}
Oct 30 22:25:06.906357 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:06.906321    4473 generic.go:356] "Generic (PLEG): container finished" podID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerID="3133aa3a8486f9bf7e6c2425a71c758a7191d5d6ffa581b7682ed6268f3d6528" exitCode=0
Oct 30 22:25:06.906796 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:06.906395    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" event={"ID":"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2","Type":"ContainerDied","Data":"3133aa3a8486f9bf7e6c2425a71c758a7191d5d6ffa581b7682ed6268f3d6528"}
Oct 30 22:25:07.912790 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:07.912751    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" event={"ID":"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2","Type":"ContainerStarted","Data":"101cea4ec2892e73934f817254602f1b5ee3914ab60e956be7f9b6fa2c58d79f"}
Oct 30 22:25:07.913295 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:07.912819    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" event={"ID":"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2","Type":"ContainerStarted","Data":"003f82a97f4144b0e4844e65dcaa8986d71b19fa35648ed1933262ce8c620a14"}
Oct 30 22:25:07.913295 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:07.913090    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:07.913295 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:07.913226    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:07.914612 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:07.914592    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 22:25:08.035501 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:08.035453    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" podStartSLOduration=3.115362114 podStartE2EDuration="7.035439271s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.920077157s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:08.034032828 +0000 UTC m=+1926.270242173" watchObservedRunningTime="2026-10-30 22:25:08.035439271 +0000 UTC m=+1926.271648644"
Oct 30 22:25:08.917138 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:08.917102    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 22:25:11.304348 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:25:11.304305    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:25:13.922528 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:13.922459    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:25:13.923005 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:13.922980    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 22:25:23.923748 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:23.923704    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 22:25:33.922888 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:33.922846    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 22:25:39.974558 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:39.974511    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:41.303422 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:25:41.303383    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:25:43.923087 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:43.923048    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 22:25:53.923233 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:25:53.923199    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 22:26:02.396203 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:02.396166    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:03.923153 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:03.923118    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 22:26:11.304042 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:26:11.304004    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:26:13.923671 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:13.923631    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 22:26:19.269586 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:19.269540    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:23.924387 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:23.924360    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:26:33.205923 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:33.205884    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"]
Oct 30 22:26:33.206369 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:33.206133    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kserve-container" containerID="cri-o://003f82a97f4144b0e4844e65dcaa8986d71b19fa35648ed1933262ce8c620a14" gracePeriod=30
Oct 30 22:26:33.206369 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:33.206176    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kube-rbac-proxy" containerID="cri-o://101cea4ec2892e73934f817254602f1b5ee3914ab60e956be7f9b6fa2c58d79f" gracePeriod=30
Oct 30 22:26:33.229238 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:33.229214    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:33.738034 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:33.737995    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:33.917693 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:33.917648    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Oct 30 22:26:33.923018 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:33.922992    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 22:26:34.267335 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:34.267299    4473 generic.go:356] "Generic (PLEG): container finished" podID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerID="101cea4ec2892e73934f817254602f1b5ee3914ab60e956be7f9b6fa2c58d79f" exitCode=0
Oct 30 22:26:34.267873 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:34.267384    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" event={"ID":"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2","Type":"ContainerDied","Data":"101cea4ec2892e73934f817254602f1b5ee3914ab60e956be7f9b6fa2c58d79f"}
Oct 30 22:26:38.128733 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.128699    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:38.287739 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.287702    4473 generic.go:356] "Generic (PLEG): container finished" podID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerID="003f82a97f4144b0e4844e65dcaa8986d71b19fa35648ed1933262ce8c620a14" exitCode=0
Oct 30 22:26:38.287910 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.287761    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" event={"ID":"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2","Type":"ContainerDied","Data":"003f82a97f4144b0e4844e65dcaa8986d71b19fa35648ed1933262ce8c620a14"}
Oct 30 22:26:38.347142 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.347119    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:26:38.403725 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.403696    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-cabundle-cert\") pod \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") "
Oct 30 22:26:38.403860 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.403741    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-proxy-tls\") pod \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") "
Oct 30 22:26:38.403860 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.403788    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kserve-provision-location\") pod \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") "
Oct 30 22:26:38.403860 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.403830    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kube-rbac-proxy-sar-config\") pod \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") "
Oct 30 22:26:38.404070 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.403861    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kube-api-access-q46v6\" (UniqueName: \"kubernetes.io/projected/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kube-api-access-q46v6\") pod \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\" (UID: \"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2\") "
Oct 30 22:26:38.404154 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.404130    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kserve-provision-location" pod "a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" (UID: "a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:38.404215 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.404130    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-cabundle-cert" pod "a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" (UID: "a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:38.404274 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.404234    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kube-rbac-proxy-sar-config" pod "a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" (UID: "a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:38.406049 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.406020    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-proxy-tls" pod "a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" (UID: "a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:38.406147 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.406083    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kube-api-access-q46v6" pod "a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" (UID: "a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2"). InnerVolumeSpecName "kube-api-access-q46v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:38.504611 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.504577    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:38.504611 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.504605    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:38.504611 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.504616    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:38.504854 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.504626    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:38.504854 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.504635    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q46v6\" (UniqueName: \"kubernetes.io/projected/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2-kube-api-access-q46v6\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:38.539941 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.539901    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:38.843018 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.842923    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"]
Oct 30 22:26:38.843261 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.843248    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="storage-initializer"
Oct 30 22:26:38.843301 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.843263    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="storage-initializer"
Oct 30 22:26:38.843301 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.843273    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kube-rbac-proxy"
Oct 30 22:26:38.843301 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.843278    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kube-rbac-proxy"
Oct 30 22:26:38.843301 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.843296    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kserve-container"
Oct 30 22:26:38.843424 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.843302    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kserve-container"
Oct 30 22:26:38.843424 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.843353    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kube-rbac-proxy"
Oct 30 22:26:38.843424 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.843364    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" containerName="kserve-container"
Oct 30 22:26:38.846917 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.846897    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:26:38.900394 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.900363    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:26:38.900552 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.900422    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"kube-root-ca.crt\""
Oct 30 22:26:38.900552 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.900492    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"openshift-service-ca.crt\""
Oct 30 22:26:38.908178 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.908144    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:26:38.908313 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.908191    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:26:38.908313 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.908297    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-proxy-tls\") pod \"isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:26:38.908442 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.908325    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:26:38.908442 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.908414    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttwqh\" (UniqueName: \"kubernetes.io/projected/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kube-api-access-ttwqh\") pod \"isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:26:38.951156 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.951131    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"seaweedfs-s3-creds\""
Oct 30 22:26:38.951156 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.951145    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Oct 30 22:26:38.951156 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.951131    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Oct 30 22:26:38.951156 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.951165    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"default-dockercfg-4v4s8\""
Oct 30 22:26:38.952733 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:38.952714    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"]
Oct 30 22:26:39.009681 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.009639    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:26:39.009681 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.009685    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:26:39.009921 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.009715    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-proxy-tls\") pod \"isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:26:39.009921 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.009733    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:26:39.009921 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.009763    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttwqh\" (UniqueName: \"kubernetes.io/projected/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kube-api-access-ttwqh\") pod \"isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:26:39.010246 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.010222    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:26:39.010421 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.010404    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:26:39.010481 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.010439    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:26:39.012273 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.012255    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-proxy-tls\") pod \"isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:26:39.068081 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.068052    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttwqh\" (UniqueName: \"kubernetes.io/projected/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kube-api-access-ttwqh\") pod \"isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:26:39.163677 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.163643    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:26:39.293705 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.293681    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"
Oct 30 22:26:39.293877 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.293676    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt" event={"ID":"a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2","Type":"ContainerDied","Data":"cd63ccfc41dab82deb21b6e2bfe16989acf398654f6cf24816c3da771fbe71c6"}
Oct 30 22:26:39.293877 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.293824    4473 scope.go:122] "RemoveContainer" containerID="101cea4ec2892e73934f817254602f1b5ee3914ab60e956be7f9b6fa2c58d79f"
Oct 30 22:26:39.301965 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.301927    4473 scope.go:122] "RemoveContainer" containerID="003f82a97f4144b0e4844e65dcaa8986d71b19fa35648ed1933262ce8c620a14"
Oct 30 22:26:39.309500 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.309482    4473 scope.go:122] "RemoveContainer" containerID="3133aa3a8486f9bf7e6c2425a71c758a7191d5d6ffa581b7682ed6268f3d6528"
Oct 30 22:26:39.460238 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.460164    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"]
Oct 30 22:26:39.463339 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.463315    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-predictor-7db6554897-vrnjt"]
Oct 30 22:26:39.578209 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.578186    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"]
Oct 30 22:26:39.580583 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:26:39.580549    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996d5c6e_bd2d_430b_9b0d_c75513bb65bd.slice/crio-067f67724a28e5dee4e55bd76e928e85bbd6e315029e130a4a8b0a22ed5767c1 WatchSource:0}: Error finding container 067f67724a28e5dee4e55bd76e928e85bbd6e315029e130a4a8b0a22ed5767c1: Status 404 returned error can't find the container with id 067f67724a28e5dee4e55bd76e928e85bbd6e315029e130a4a8b0a22ed5767c1
Oct 30 22:26:39.885714 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:39.885623    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2" path="/var/lib/kubelet/pods/a6e46be5-ef5b-44a2-bd61-bdf6ca28f7c2/volumes"
Oct 30 22:26:40.299224 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:40.299189    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7" event={"ID":"996d5c6e-bd2d-430b-9b0d-c75513bb65bd","Type":"ContainerStarted","Data":"658d477f247c2506f6bdecefc6ae0b6c88429359e627c3a24983b6c852235294"}
Oct 30 22:26:40.299652 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:40.299254    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7" event={"ID":"996d5c6e-bd2d-430b-9b0d-c75513bb65bd","Type":"ContainerStarted","Data":"067f67724a28e5dee4e55bd76e928e85bbd6e315029e130a4a8b0a22ed5767c1"}
Oct 30 22:26:41.303927 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:26:41.303895    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:26:42.114718 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.114677    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:42.368441 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.368336    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:42.621255 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.621179    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"]
Oct 30 22:26:42.624901 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.624884    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:42.638128 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.638106    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:42.638224 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.638139    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:42.638267 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.638228    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:42.638267 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.638256    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t9jq\" (UniqueName: \"kubernetes.io/projected/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kube-api-access-9t9jq\") pod \"isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:42.638353 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.638285    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:42.678524 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.678492    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Oct 30 22:26:42.728856 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.728815    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:26:42.729043 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.728926    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:26:42.729043 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.728936    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:26:42.729043 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.728973    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:26:42.729043 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.728987    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:26:42.729043 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.729023    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Oct 30 22:26:42.730530 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.730504    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"]
Oct 30 22:26:42.738980 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.738933    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:42.739109 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.738994    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:42.739109 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.739074    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:42.739109 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.739104    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t9jq\" (UniqueName: \"kubernetes.io/projected/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kube-api-access-9t9jq\") pod \"isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:42.739275 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.739217    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:42.739516 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.739498    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:42.739887 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.739869    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:42.739930 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.739880    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:42.741652 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.741631    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:42.797307 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.797273    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t9jq\" (UniqueName: \"kubernetes.io/projected/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kube-api-access-9t9jq\") pod \"isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") " pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:42.940873 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:42.940841    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:43.415544 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:43.415463    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"]
Oct 30 22:26:43.417758 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:26:43.417730    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3339d3a8_ad3d_498d_a9b1_bb07a7038f4d.slice/crio-9243b389b3ec464617c374130fce55f56c4cd654bfb1d6866d53bd64dad0ac54 WatchSource:0}: Error finding container 9243b389b3ec464617c374130fce55f56c4cd654bfb1d6866d53bd64dad0ac54: Status 404 returned error can't find the container with id 9243b389b3ec464617c374130fce55f56c4cd654bfb1d6866d53bd64dad0ac54
Oct 30 22:26:44.318105 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:44.318069    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" event={"ID":"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d","Type":"ContainerStarted","Data":"7d41eefd57854b7359e4e751455751a9d0a9433d848889249ca312a7765f8ff2"}
Oct 30 22:26:44.318295 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:44.318144    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" event={"ID":"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d","Type":"ContainerStarted","Data":"9243b389b3ec464617c374130fce55f56c4cd654bfb1d6866d53bd64dad0ac54"}
Oct 30 22:26:48.335605 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:48.335564    4473 generic.go:356] "Generic (PLEG): container finished" podID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerID="7d41eefd57854b7359e4e751455751a9d0a9433d848889249ca312a7765f8ff2" exitCode=0
Oct 30 22:26:48.336021 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:48.335639    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" event={"ID":"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d","Type":"ContainerDied","Data":"7d41eefd57854b7359e4e751455751a9d0a9433d848889249ca312a7765f8ff2"}
Oct 30 22:26:49.342170 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:49.342131    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" event={"ID":"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d","Type":"ContainerStarted","Data":"395f2334b0fc6ce84fc7c216cae117a8c02601cddbb864eb1061948f9e357ae0"}
Oct 30 22:26:49.342670 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:49.342203    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" event={"ID":"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d","Type":"ContainerStarted","Data":"5fd9591f2885a79376ebb979b1d188d604f344e0b4f266ba9428a1a3bd09481e"}
Oct 30 22:26:49.342670 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:49.342529    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:49.513081 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:49.513017    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" podStartSLOduration=3.434468872 podStartE2EDuration="7.513001801s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.078532929s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:49.462055258 +0000 UTC m=+2027.698264603" watchObservedRunningTime="2026-10-30 22:26:49.513001801 +0000 UTC m=+2027.749211145"
Oct 30 22:26:50.345834 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:50.345799    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:50.347185 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:50.347159    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Oct 30 22:26:51.349596 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:51.349561    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Oct 30 22:26:56.354741 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:56.354704    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:26:56.355404 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:26:56.355375    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Oct 30 22:27:02.774528 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:27:02.774490    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:06.355561 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:27:06.355524    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Oct 30 22:27:11.303330 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:27:11.303288    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:27:11.432895 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:27:11.432864    4473 generic.go:356] "Generic (PLEG): container finished" podID="996d5c6e-bd2d-430b-9b0d-c75513bb65bd" containerID="658d477f247c2506f6bdecefc6ae0b6c88429359e627c3a24983b6c852235294" exitCode=0
Oct 30 22:27:11.433055 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:27:11.432937    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7" event={"ID":"996d5c6e-bd2d-430b-9b0d-c75513bb65bd","Type":"ContainerDied","Data":"658d477f247c2506f6bdecefc6ae0b6c88429359e627c3a24983b6c852235294"}
Oct 30 22:27:11.434378 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:27:11.434363    4473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:27:16.355628 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:27:16.355581    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Oct 30 22:27:26.355451 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:27:26.355409    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Oct 30 22:27:36.355409 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:27:36.355359    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Oct 30 22:27:41.309875 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:27:41.309839    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:27:46.356129 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:27:46.356085    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Oct 30 22:27:56.355307 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:27:56.355261    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Oct 30 22:28:01.885019 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:01.884903    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Oct 30 22:28:02.071635 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:02.071603    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:28:02.074409 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:02.074383    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:28:03.198219 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:03.198176    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:11.308296 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:28:11.308244    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:28:11.885102 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:11.885055    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Oct 30 22:28:21.888084 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:21.888002    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:28:32.778598 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:32.778560    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:32.785795 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:32.785767    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"]
Oct 30 22:28:32.786067 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:32.786037    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kserve-container" containerID="cri-o://5fd9591f2885a79376ebb979b1d188d604f344e0b4f266ba9428a1a3bd09481e" gracePeriod=30
Oct 30 22:28:32.786201 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:32.786090    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kube-rbac-proxy" containerID="cri-o://395f2334b0fc6ce84fc7c216cae117a8c02601cddbb864eb1061948f9e357ae0" gracePeriod=30
Oct 30 22:28:33.834191 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:33.834152    4473 generic.go:356] "Generic (PLEG): container finished" podID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerID="395f2334b0fc6ce84fc7c216cae117a8c02601cddbb864eb1061948f9e357ae0" exitCode=0
Oct 30 22:28:33.834580 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:33.834232    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" event={"ID":"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d","Type":"ContainerDied","Data":"395f2334b0fc6ce84fc7c216cae117a8c02601cddbb864eb1061948f9e357ae0"}
Oct 30 22:28:36.350636 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:36.350595    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Oct 30 22:28:38.459627 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.459592    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:28:38.528213 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.528124    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kserve-provision-location\") pod \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") "
Oct 30 22:28:38.528392 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.528215    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kube-api-access-9t9jq\" (UniqueName: \"kubernetes.io/projected/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kube-api-access-9t9jq\") pod \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") "
Oct 30 22:28:38.528392 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.528280    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-proxy-tls\") pod \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") "
Oct 30 22:28:38.528392 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.528324    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kube-rbac-proxy-sar-config\") pod \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") "
Oct 30 22:28:38.528392 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.528383    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-cabundle-cert\") pod \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\" (UID: \"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d\") "
Oct 30 22:28:38.528594 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.528427    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kserve-provision-location" pod "3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" (UID: "3339d3a8-ad3d-498d-a9b1-bb07a7038f4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:38.528646 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.528595    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:38.529056 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.529022    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-cabundle-cert" pod "3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" (UID: "3339d3a8-ad3d-498d-a9b1-bb07a7038f4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:38.529191 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.529091    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kube-rbac-proxy-sar-config" pod "3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" (UID: "3339d3a8-ad3d-498d-a9b1-bb07a7038f4d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:38.531042 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.531012    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-proxy-tls" pod "3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" (UID: "3339d3a8-ad3d-498d-a9b1-bb07a7038f4d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:38.531261 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.531235    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kube-api-access-9t9jq" pod "3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" (UID: "3339d3a8-ad3d-498d-a9b1-bb07a7038f4d"). InnerVolumeSpecName "kube-api-access-9t9jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:38.630057 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.630013    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:38.630057 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.630045    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:38.630388 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.630085    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:38.630388 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.630095    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t9jq\" (UniqueName: \"kubernetes.io/projected/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d-kube-api-access-9t9jq\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:38.860008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.859886    4473 generic.go:356] "Generic (PLEG): container finished" podID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerID="5fd9591f2885a79376ebb979b1d188d604f344e0b4f266ba9428a1a3bd09481e" exitCode=0
Oct 30 22:28:38.860008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.859993    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"
Oct 30 22:28:38.860008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.859988    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" event={"ID":"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d","Type":"ContainerDied","Data":"5fd9591f2885a79376ebb979b1d188d604f344e0b4f266ba9428a1a3bd09481e"}
Oct 30 22:28:38.860391 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.860100    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd" event={"ID":"3339d3a8-ad3d-498d-a9b1-bb07a7038f4d","Type":"ContainerDied","Data":"9243b389b3ec464617c374130fce55f56c4cd654bfb1d6866d53bd64dad0ac54"}
Oct 30 22:28:38.860391 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.860137    4473 scope.go:122] "RemoveContainer" containerID="395f2334b0fc6ce84fc7c216cae117a8c02601cddbb864eb1061948f9e357ae0"
Oct 30 22:28:38.870083 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.870067    4473 scope.go:122] "RemoveContainer" containerID="5fd9591f2885a79376ebb979b1d188d604f344e0b4f266ba9428a1a3bd09481e"
Oct 30 22:28:38.878041 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.878021    4473 scope.go:122] "RemoveContainer" containerID="7d41eefd57854b7359e4e751455751a9d0a9433d848889249ca312a7765f8ff2"
Oct 30 22:28:38.885803 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.885784    4473 scope.go:122] "RemoveContainer" containerID="395f2334b0fc6ce84fc7c216cae117a8c02601cddbb864eb1061948f9e357ae0"
Oct 30 22:28:38.886246 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:28:38.886123    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"395f2334b0fc6ce84fc7c216cae117a8c02601cddbb864eb1061948f9e357ae0\": container with ID starting with 395f2334b0fc6ce84fc7c216cae117a8c02601cddbb864eb1061948f9e357ae0 not found: ID does not exist" containerID="395f2334b0fc6ce84fc7c216cae117a8c02601cddbb864eb1061948f9e357ae0"
Oct 30 22:28:38.886246 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.886162    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395f2334b0fc6ce84fc7c216cae117a8c02601cddbb864eb1061948f9e357ae0"} err="failed to get container status \"395f2334b0fc6ce84fc7c216cae117a8c02601cddbb864eb1061948f9e357ae0\": rpc error: code = NotFound desc = could not find container \"395f2334b0fc6ce84fc7c216cae117a8c02601cddbb864eb1061948f9e357ae0\": container with ID starting with 395f2334b0fc6ce84fc7c216cae117a8c02601cddbb864eb1061948f9e357ae0 not found: ID does not exist"
Oct 30 22:28:38.886246 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.886183    4473 scope.go:122] "RemoveContainer" containerID="5fd9591f2885a79376ebb979b1d188d604f344e0b4f266ba9428a1a3bd09481e"
Oct 30 22:28:38.886484 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:28:38.886463    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fd9591f2885a79376ebb979b1d188d604f344e0b4f266ba9428a1a3bd09481e\": container with ID starting with 5fd9591f2885a79376ebb979b1d188d604f344e0b4f266ba9428a1a3bd09481e not found: ID does not exist" containerID="5fd9591f2885a79376ebb979b1d188d604f344e0b4f266ba9428a1a3bd09481e"
Oct 30 22:28:38.886524 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.886494    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd9591f2885a79376ebb979b1d188d604f344e0b4f266ba9428a1a3bd09481e"} err="failed to get container status \"5fd9591f2885a79376ebb979b1d188d604f344e0b4f266ba9428a1a3bd09481e\": rpc error: code = NotFound desc = could not find container \"5fd9591f2885a79376ebb979b1d188d604f344e0b4f266ba9428a1a3bd09481e\": container with ID starting with 5fd9591f2885a79376ebb979b1d188d604f344e0b4f266ba9428a1a3bd09481e not found: ID does not exist"
Oct 30 22:28:38.886524 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.886516    4473 scope.go:122] "RemoveContainer" containerID="7d41eefd57854b7359e4e751455751a9d0a9433d848889249ca312a7765f8ff2"
Oct 30 22:28:38.886793 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:28:38.886774    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d41eefd57854b7359e4e751455751a9d0a9433d848889249ca312a7765f8ff2\": container with ID starting with 7d41eefd57854b7359e4e751455751a9d0a9433d848889249ca312a7765f8ff2 not found: ID does not exist" containerID="7d41eefd57854b7359e4e751455751a9d0a9433d848889249ca312a7765f8ff2"
Oct 30 22:28:38.886877 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:38.886797    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d41eefd57854b7359e4e751455751a9d0a9433d848889249ca312a7765f8ff2"} err="failed to get container status \"7d41eefd57854b7359e4e751455751a9d0a9433d848889249ca312a7765f8ff2\": rpc error: code = NotFound desc = could not find container \"7d41eefd57854b7359e4e751455751a9d0a9433d848889249ca312a7765f8ff2\": container with ID starting with 7d41eefd57854b7359e4e751455751a9d0a9433d848889249ca312a7765f8ff2 not found: ID does not exist"
Oct 30 22:28:39.026961 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:39.026918    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"]
Oct 30 22:28:39.077202 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:39.077158    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-v2-mixed-predictor-66b9d9576-x9qmd"]
Oct 30 22:28:39.887057 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:39.887021    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" path="/var/lib/kubelet/pods/3339d3a8-ad3d-498d-a9b1-bb07a7038f4d/volumes"
Oct 30 22:28:41.062435 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:41.062396    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:41.309081 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:28:41.309037    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:28:41.348030 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:41.347938    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:59.961638 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:59.961600    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7" event={"ID":"996d5c6e-bd2d-430b-9b0d-c75513bb65bd","Type":"ContainerStarted","Data":"cab245ba2fd6771218f27b9a9065adbb53a43f557c538b119b53e699bf082a53"}
Oct 30 22:28:59.962186 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:59.961667    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7" event={"ID":"996d5c6e-bd2d-430b-9b0d-c75513bb65bd","Type":"ContainerStarted","Data":"fb443cf4b71aba51870639112ae22930acdde5058c6b3941db092523ad7732b3"}
Oct 30 22:28:59.962186 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:28:59.961748    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:29:00.133063 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:29:00.133004    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7" podStartSLOduration=2.603490239 podStartE2EDuration="2m22.132988549s" totalImagesPullingTime="1m47.775524268s" totalInitContainerRuntime="31.753974042s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:00.132502186 +0000 UTC m=+2158.368711542" watchObservedRunningTime="2026-10-30 22:29:00.132988549 +0000 UTC m=+2158.369197898"
Oct 30 22:29:00.966246 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:29:00.966214    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:29:03.472369 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:29:03.472334    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:06.976688 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:29:06.976655    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:29:11.303460 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:29:11.303423    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:29:32.451220 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:29:32.451186    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:41.303490 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:29:41.303451    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:30:03.776877 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:03.776847    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:07.458115 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:07.458076    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:07.692612 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:07.692578    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:11.303057 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:30:11.303015    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:30:35.914787 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:35.914750    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:30:41.304765 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:30:41.304729    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:30:45.221337 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:45.221307    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"]
Oct 30 22:30:45.221735 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:45.221561    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7" podUID="996d5c6e-bd2d-430b-9b0d-c75513bb65bd" containerName="kserve-container" containerID="cri-o://fb443cf4b71aba51870639112ae22930acdde5058c6b3941db092523ad7732b3" gracePeriod=30
Oct 30 22:30:45.221735 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:45.221672    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7" podUID="996d5c6e-bd2d-430b-9b0d-c75513bb65bd" containerName="kube-rbac-proxy" containerID="cri-o://cab245ba2fd6771218f27b9a9065adbb53a43f557c538b119b53e699bf082a53" gracePeriod=30
Oct 30 22:30:45.278398 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:45.278365    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:45.376679 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:45.376648    4473 generic.go:356] "Generic (PLEG): container finished" podID="996d5c6e-bd2d-430b-9b0d-c75513bb65bd" containerID="cab245ba2fd6771218f27b9a9065adbb53a43f557c538b119b53e699bf082a53" exitCode=0
Oct 30 22:30:45.376845 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:45.376722    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7" event={"ID":"996d5c6e-bd2d-430b-9b0d-c75513bb65bd","Type":"ContainerDied","Data":"cab245ba2fd6771218f27b9a9065adbb53a43f557c538b119b53e699bf082a53"}
Oct 30 22:30:45.881705 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:45.881664    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7" podUID="996d5c6e-bd2d-430b-9b0d-c75513bb65bd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.46:8080/v2/models/isvc-autogluon-v2/ready\": dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 30 22:30:46.665573 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:46.665547    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:30:46.707141 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:46.707110    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kube-api-access-ttwqh\" (UniqueName: \"kubernetes.io/projected/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kube-api-access-ttwqh\") pod \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") "
Oct 30 22:30:46.707241 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:46.707166    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kube-rbac-proxy-sar-config\") pod \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") "
Oct 30 22:30:46.707241 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:46.707196    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-proxy-tls\" (UniqueName: \"kubernetes.io/secret/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-proxy-tls\") pod \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") "
Oct 30 22:30:46.707241 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:46.707220    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kserve-provision-location\") pod \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") "
Oct 30 22:30:46.707367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:46.707248    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-cabundle-cert\") pod \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\" (UID: \"996d5c6e-bd2d-430b-9b0d-c75513bb65bd\") "
Oct 30 22:30:46.707566 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:46.707536    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kube-rbac-proxy-sar-config" pod "996d5c6e-bd2d-430b-9b0d-c75513bb65bd" (UID: "996d5c6e-bd2d-430b-9b0d-c75513bb65bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:46.707716 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:46.707686    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-cabundle-cert" pod "996d5c6e-bd2d-430b-9b0d-c75513bb65bd" (UID: "996d5c6e-bd2d-430b-9b0d-c75513bb65bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:46.709430 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:46.709406    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kube-api-access-ttwqh" pod "996d5c6e-bd2d-430b-9b0d-c75513bb65bd" (UID: "996d5c6e-bd2d-430b-9b0d-c75513bb65bd"). InnerVolumeSpecName "kube-api-access-ttwqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:46.709430 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:46.709413    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-proxy-tls" pod "996d5c6e-bd2d-430b-9b0d-c75513bb65bd" (UID: "996d5c6e-bd2d-430b-9b0d-c75513bb65bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:46.710775 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:46.710753    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kserve-provision-location" pod "996d5c6e-bd2d-430b-9b0d-c75513bb65bd" (UID: "996d5c6e-bd2d-430b-9b0d-c75513bb65bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:46.808868 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:46.808784    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:46.808868 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:46.808815    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttwqh\" (UniqueName: \"kubernetes.io/projected/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kube-api-access-ttwqh\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:46.808868 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:46.808826    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:46.808868 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:46.808835    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:46.808868 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:46.808851    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996d5c6e-bd2d-430b-9b0d-c75513bb65bd-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:47.385847 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:47.385813    4473 generic.go:356] "Generic (PLEG): container finished" podID="996d5c6e-bd2d-430b-9b0d-c75513bb65bd" containerID="fb443cf4b71aba51870639112ae22930acdde5058c6b3941db092523ad7732b3" exitCode=0
Oct 30 22:30:47.386041 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:47.385862    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7" event={"ID":"996d5c6e-bd2d-430b-9b0d-c75513bb65bd","Type":"ContainerDied","Data":"fb443cf4b71aba51870639112ae22930acdde5058c6b3941db092523ad7732b3"}
Oct 30 22:30:47.386041 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:47.385881    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"
Oct 30 22:30:47.386041 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:47.385910    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7" event={"ID":"996d5c6e-bd2d-430b-9b0d-c75513bb65bd","Type":"ContainerDied","Data":"067f67724a28e5dee4e55bd76e928e85bbd6e315029e130a4a8b0a22ed5767c1"}
Oct 30 22:30:47.386041 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:47.385936    4473 scope.go:122] "RemoveContainer" containerID="cab245ba2fd6771218f27b9a9065adbb53a43f557c538b119b53e699bf082a53"
Oct 30 22:30:47.394387 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:47.394369    4473 scope.go:122] "RemoveContainer" containerID="fb443cf4b71aba51870639112ae22930acdde5058c6b3941db092523ad7732b3"
Oct 30 22:30:47.401814 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:47.401793    4473 scope.go:122] "RemoveContainer" containerID="658d477f247c2506f6bdecefc6ae0b6c88429359e627c3a24983b6c852235294"
Oct 30 22:30:47.409295 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:47.409279    4473 scope.go:122] "RemoveContainer" containerID="cab245ba2fd6771218f27b9a9065adbb53a43f557c538b119b53e699bf082a53"
Oct 30 22:30:47.409561 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:30:47.409543    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab245ba2fd6771218f27b9a9065adbb53a43f557c538b119b53e699bf082a53\": container with ID starting with cab245ba2fd6771218f27b9a9065adbb53a43f557c538b119b53e699bf082a53 not found: ID does not exist" containerID="cab245ba2fd6771218f27b9a9065adbb53a43f557c538b119b53e699bf082a53"
Oct 30 22:30:47.409601 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:47.409571    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab245ba2fd6771218f27b9a9065adbb53a43f557c538b119b53e699bf082a53"} err="failed to get container status \"cab245ba2fd6771218f27b9a9065adbb53a43f557c538b119b53e699bf082a53\": rpc error: code = NotFound desc = could not find container \"cab245ba2fd6771218f27b9a9065adbb53a43f557c538b119b53e699bf082a53\": container with ID starting with cab245ba2fd6771218f27b9a9065adbb53a43f557c538b119b53e699bf082a53 not found: ID does not exist"
Oct 30 22:30:47.409601 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:47.409588    4473 scope.go:122] "RemoveContainer" containerID="fb443cf4b71aba51870639112ae22930acdde5058c6b3941db092523ad7732b3"
Oct 30 22:30:47.409818 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:30:47.409802    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb443cf4b71aba51870639112ae22930acdde5058c6b3941db092523ad7732b3\": container with ID starting with fb443cf4b71aba51870639112ae22930acdde5058c6b3941db092523ad7732b3 not found: ID does not exist" containerID="fb443cf4b71aba51870639112ae22930acdde5058c6b3941db092523ad7732b3"
Oct 30 22:30:47.409854 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:47.409827    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb443cf4b71aba51870639112ae22930acdde5058c6b3941db092523ad7732b3"} err="failed to get container status \"fb443cf4b71aba51870639112ae22930acdde5058c6b3941db092523ad7732b3\": rpc error: code = NotFound desc = could not find container \"fb443cf4b71aba51870639112ae22930acdde5058c6b3941db092523ad7732b3\": container with ID starting with fb443cf4b71aba51870639112ae22930acdde5058c6b3941db092523ad7732b3 not found: ID does not exist"
Oct 30 22:30:47.409854 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:47.409842    4473 scope.go:122] "RemoveContainer" containerID="658d477f247c2506f6bdecefc6ae0b6c88429359e627c3a24983b6c852235294"
Oct 30 22:30:47.410113 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:30:47.410095    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658d477f247c2506f6bdecefc6ae0b6c88429359e627c3a24983b6c852235294\": container with ID starting with 658d477f247c2506f6bdecefc6ae0b6c88429359e627c3a24983b6c852235294 not found: ID does not exist" containerID="658d477f247c2506f6bdecefc6ae0b6c88429359e627c3a24983b6c852235294"
Oct 30 22:30:47.410152 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:47.410118    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658d477f247c2506f6bdecefc6ae0b6c88429359e627c3a24983b6c852235294"} err="failed to get container status \"658d477f247c2506f6bdecefc6ae0b6c88429359e627c3a24983b6c852235294\": rpc error: code = NotFound desc = could not find container \"658d477f247c2506f6bdecefc6ae0b6c88429359e627c3a24983b6c852235294\": container with ID starting with 658d477f247c2506f6bdecefc6ae0b6c88429359e627c3a24983b6c852235294 not found: ID does not exist"
Oct 30 22:30:47.602280 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:47.602247    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"]
Oct 30 22:30:47.653086 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:47.653056    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-predictor-7f77dd89f8-sm8r7"]
Oct 30 22:30:47.885616 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:47.885581    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="996d5c6e-bd2d-430b-9b0d-c75513bb65bd" path="/var/lib/kubelet/pods/996d5c6e-bd2d-430b-9b0d-c75513bb65bd/volumes"
Oct 30 22:30:48.509746 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:48.509687    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:49.523653 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:49.523623    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:49.861754 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:49.861660    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:50.022373 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.022341    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"]
Oct 30 22:30:50.022641 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.022629    4473 cpu_manager.go:414] "containerMap: removing container" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kserve-container"
Oct 30 22:30:50.022698 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.022643    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kserve-container"
Oct 30 22:30:50.022698 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.022653    4473 cpu_manager.go:414] "containerMap: removing container" podUID="996d5c6e-bd2d-430b-9b0d-c75513bb65bd" containerName="kube-rbac-proxy"
Oct 30 22:30:50.022698 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.022658    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996d5c6e-bd2d-430b-9b0d-c75513bb65bd" containerName="kube-rbac-proxy"
Oct 30 22:30:50.022698 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.022670    4473 cpu_manager.go:414] "containerMap: removing container" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="storage-initializer"
Oct 30 22:30:50.022698 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.022675    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="storage-initializer"
Oct 30 22:30:50.022698 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.022684    4473 cpu_manager.go:414] "containerMap: removing container" podUID="996d5c6e-bd2d-430b-9b0d-c75513bb65bd" containerName="storage-initializer"
Oct 30 22:30:50.022698 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.022688    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996d5c6e-bd2d-430b-9b0d-c75513bb65bd" containerName="storage-initializer"
Oct 30 22:30:50.022698 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.022695    4473 cpu_manager.go:414] "containerMap: removing container" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kube-rbac-proxy"
Oct 30 22:30:50.022698 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.022701    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kube-rbac-proxy"
Oct 30 22:30:50.023029 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.022719    4473 cpu_manager.go:414] "containerMap: removing container" podUID="996d5c6e-bd2d-430b-9b0d-c75513bb65bd" containerName="kserve-container"
Oct 30 22:30:50.023029 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.022724    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996d5c6e-bd2d-430b-9b0d-c75513bb65bd" containerName="kserve-container"
Oct 30 22:30:50.023029 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.022800    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kserve-container"
Oct 30 22:30:50.023029 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.022807    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="996d5c6e-bd2d-430b-9b0d-c75513bb65bd" containerName="kube-rbac-proxy"
Oct 30 22:30:50.023029 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.022815    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3339d3a8-ad3d-498d-a9b1-bb07a7038f4d" containerName="kube-rbac-proxy"
Oct 30 22:30:50.023029 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.022824    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="996d5c6e-bd2d-430b-9b0d-c75513bb65bd" containerName="kserve-container"
Oct 30 22:30:50.025887 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.025869    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:30:50.130062 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.129976    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"seaweedfs-s3-creds\""
Oct 30 22:30:50.130062 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.129974    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"default-dockercfg-4v4s8\""
Oct 30 22:30:50.130062 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.129974    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"openshift-service-ca.crt\""
Oct 30 22:30:50.130062 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.129974    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Oct 30 22:30:50.130062 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.130036    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:30:50.130062 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.130005    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Oct 30 22:30:50.130472 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.130128    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"kube-root-ca.crt\""
Oct 30 22:30:50.131648 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.131621    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"]
Oct 30 22:30:50.135458 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.135437    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j7dl\" (UniqueName: \"kubernetes.io/projected/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kube-api-access-7j7dl\") pod \"isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:30:50.135559 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.135486    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:30:50.135559 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.135545    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:30:50.135649 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.135576    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:30:50.135649 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.135604    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:30:50.236960 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.236916    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:30:50.237138 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.236971    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:30:50.237138 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.237007    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:30:50.237138 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.237036    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:30:50.237138 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.237067    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j7dl\" (UniqueName: \"kubernetes.io/projected/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kube-api-access-7j7dl\") pod \"isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:30:50.237515 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.237494    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:30:50.237735 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.237713    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:30:50.237809 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.237788    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:30:50.239503 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.239476    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:30:50.295512 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.295484    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j7dl\" (UniqueName: \"kubernetes.io/projected/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kube-api-access-7j7dl\") pod \"isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:30:50.338803 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.338760    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:30:50.763648 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:50.763524    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"]
Oct 30 22:30:50.765866 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:30:50.765835    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f47caa_b86a_4dd1_a340_37bfc5b2e36b.slice/crio-7e22ce68d8f234b96458968b3b7e82deab4251508a43f9da0bb296a495e34449 WatchSource:0}: Error finding container 7e22ce68d8f234b96458968b3b7e82deab4251508a43f9da0bb296a495e34449: Status 404 returned error can't find the container with id 7e22ce68d8f234b96458968b3b7e82deab4251508a43f9da0bb296a495e34449
Oct 30 22:30:51.407305 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:51.407265    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg" event={"ID":"60f47caa-b86a-4dd1-a340-37bfc5b2e36b","Type":"ContainerStarted","Data":"108d96e7adf340e43e5f9146e3353c67952fff42d1e7aa838574209ba842be2e"}
Oct 30 22:30:51.407457 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:30:51.407347    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg" event={"ID":"60f47caa-b86a-4dd1-a340-37bfc5b2e36b","Type":"ContainerStarted","Data":"7e22ce68d8f234b96458968b3b7e82deab4251508a43f9da0bb296a495e34449"}
Oct 30 22:31:04.133279 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:31:04.133240    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:11.303531 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:31:11.303493    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:31:23.533348 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:31:23.533315    4473 generic.go:356] "Generic (PLEG): container finished" podID="60f47caa-b86a-4dd1-a340-37bfc5b2e36b" containerID="108d96e7adf340e43e5f9146e3353c67952fff42d1e7aa838574209ba842be2e" exitCode=0
Oct 30 22:31:23.533729 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:31:23.533398    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg" event={"ID":"60f47caa-b86a-4dd1-a340-37bfc5b2e36b","Type":"ContainerDied","Data":"108d96e7adf340e43e5f9146e3353c67952fff42d1e7aa838574209ba842be2e"}
Oct 30 22:31:23.600068 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:31:23.600037    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:23.907204 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:31:23.907172    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:24.540607 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:31:24.540575    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg" event={"ID":"60f47caa-b86a-4dd1-a340-37bfc5b2e36b","Type":"ContainerStarted","Data":"3a2115f66e34c73a59bf11788627de1ecb6e49f5c3e88efb39521b87dbc16658"}
Oct 30 22:31:24.541021 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:31:24.540645    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg" event={"ID":"60f47caa-b86a-4dd1-a340-37bfc5b2e36b","Type":"ContainerStarted","Data":"e27c816272f81b7af973f138468330bfa3320ee4dfe39dab9e899d5518fe2f5a"}
Oct 30 22:31:24.541021 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:31:24.540839    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:31:24.714404 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:31:24.714352    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg" podStartSLOduration=3.161944866 podStartE2EDuration="35.714336809s" totalImagesPullingTime="0s" totalInitContainerRuntime="32.552391943s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:24.713192589 +0000 UTC m=+2302.949401935" watchObservedRunningTime="2026-10-30 22:31:24.714336809 +0000 UTC m=+2302.950546154"
Oct 30 22:31:25.545178 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:31:25.545149    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:31:31.556297 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:31:31.556269    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:31:40.154466 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:31:40.154426    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:41.303752 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:31:41.303716    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:31:44.450205 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:31:44.450130    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:44.611847 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:31:44.611811    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:04.449655 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:32:04.449598    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:11.305102 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:32:11.305064    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:32:41.303906 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:32:41.303869    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:32:51.935253 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:32:51.935158    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:00.913992 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:00.913942    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:33:02.099301 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:02.099260    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:33:02.102428 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:02.102406    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:33:04.792766 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:04.792737    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:11.304322 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:33:11.304279    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:33:11.923154 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:11.923111    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg" podUID="60f47caa-b86a-4dd1-a340-37bfc5b2e36b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.48:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:33:17.744002 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:17.743941    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:17.762079 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:17.762050    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"]
Oct 30 22:33:17.762308 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:17.762276    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg" podUID="60f47caa-b86a-4dd1-a340-37bfc5b2e36b" containerName="kserve-container" containerID="cri-o://e27c816272f81b7af973f138468330bfa3320ee4dfe39dab9e899d5518fe2f5a" gracePeriod=30
Oct 30 22:33:17.762430 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:17.762318    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg" podUID="60f47caa-b86a-4dd1-a340-37bfc5b2e36b" containerName="kube-rbac-proxy" containerID="cri-o://3a2115f66e34c73a59bf11788627de1ecb6e49f5c3e88efb39521b87dbc16658" gracePeriod=30
Oct 30 22:33:18.002219 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:18.002124    4473 generic.go:356] "Generic (PLEG): container finished" podID="60f47caa-b86a-4dd1-a340-37bfc5b2e36b" containerID="3a2115f66e34c73a59bf11788627de1ecb6e49f5c3e88efb39521b87dbc16658" exitCode=0
Oct 30 22:33:18.002219 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:18.002186    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg" event={"ID":"60f47caa-b86a-4dd1-a340-37bfc5b2e36b","Type":"ContainerDied","Data":"3a2115f66e34c73a59bf11788627de1ecb6e49f5c3e88efb39521b87dbc16658"}
Oct 30 22:33:19.199208 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:19.199185    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:33:19.292020 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:19.291916    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kube-rbac-proxy-sar-config\") pod \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") "
Oct 30 22:33:19.292020 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:19.291997    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kserve-provision-location\") pod \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") "
Oct 30 22:33:19.292265 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:19.292035    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kube-api-access-7j7dl\" (UniqueName: \"kubernetes.io/projected/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kube-api-access-7j7dl\") pod \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") "
Oct 30 22:33:19.292265 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:19.292053    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-cabundle-cert\") pod \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") "
Oct 30 22:33:19.292265 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:19.292104    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-proxy-tls\") pod \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\" (UID: \"60f47caa-b86a-4dd1-a340-37bfc5b2e36b\") "
Oct 30 22:33:19.292433 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:19.292329    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kube-rbac-proxy-sar-config" pod "60f47caa-b86a-4dd1-a340-37bfc5b2e36b" (UID: "60f47caa-b86a-4dd1-a340-37bfc5b2e36b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:19.292433 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:19.292425    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-cabundle-cert" pod "60f47caa-b86a-4dd1-a340-37bfc5b2e36b" (UID: "60f47caa-b86a-4dd1-a340-37bfc5b2e36b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:19.294405 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:19.294377    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kube-api-access-7j7dl" pod "60f47caa-b86a-4dd1-a340-37bfc5b2e36b" (UID: "60f47caa-b86a-4dd1-a340-37bfc5b2e36b"). InnerVolumeSpecName "kube-api-access-7j7dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:19.294522 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:19.294377    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-proxy-tls" pod "60f47caa-b86a-4dd1-a340-37bfc5b2e36b" (UID: "60f47caa-b86a-4dd1-a340-37bfc5b2e36b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:19.296796 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:19.296769    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kserve-provision-location" pod "60f47caa-b86a-4dd1-a340-37bfc5b2e36b" (UID: "60f47caa-b86a-4dd1-a340-37bfc5b2e36b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:19.393055 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:19.393023    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:19.393055 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:19.393049    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:19.393055 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:19.393058    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j7dl\" (UniqueName: \"kubernetes.io/projected/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-kube-api-access-7j7dl\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:19.393288 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:19.393069    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:19.393288 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:19.393078    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60f47caa-b86a-4dd1-a340-37bfc5b2e36b-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:20.011666 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:20.011626    4473 generic.go:356] "Generic (PLEG): container finished" podID="60f47caa-b86a-4dd1-a340-37bfc5b2e36b" containerID="e27c816272f81b7af973f138468330bfa3320ee4dfe39dab9e899d5518fe2f5a" exitCode=0
Oct 30 22:33:20.011847 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:20.011698    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"
Oct 30 22:33:20.011847 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:20.011714    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg" event={"ID":"60f47caa-b86a-4dd1-a340-37bfc5b2e36b","Type":"ContainerDied","Data":"e27c816272f81b7af973f138468330bfa3320ee4dfe39dab9e899d5518fe2f5a"}
Oct 30 22:33:20.011847 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:20.011778    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg" event={"ID":"60f47caa-b86a-4dd1-a340-37bfc5b2e36b","Type":"ContainerDied","Data":"7e22ce68d8f234b96458968b3b7e82deab4251508a43f9da0bb296a495e34449"}
Oct 30 22:33:20.011847 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:20.011804    4473 scope.go:122] "RemoveContainer" containerID="3a2115f66e34c73a59bf11788627de1ecb6e49f5c3e88efb39521b87dbc16658"
Oct 30 22:33:20.019986 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:20.019962    4473 scope.go:122] "RemoveContainer" containerID="e27c816272f81b7af973f138468330bfa3320ee4dfe39dab9e899d5518fe2f5a"
Oct 30 22:33:20.027503 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:20.027483    4473 scope.go:122] "RemoveContainer" containerID="108d96e7adf340e43e5f9146e3353c67952fff42d1e7aa838574209ba842be2e"
Oct 30 22:33:20.034745 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:20.034725    4473 scope.go:122] "RemoveContainer" containerID="3a2115f66e34c73a59bf11788627de1ecb6e49f5c3e88efb39521b87dbc16658"
Oct 30 22:33:20.035041 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:33:20.035018    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a2115f66e34c73a59bf11788627de1ecb6e49f5c3e88efb39521b87dbc16658\": container with ID starting with 3a2115f66e34c73a59bf11788627de1ecb6e49f5c3e88efb39521b87dbc16658 not found: ID does not exist" containerID="3a2115f66e34c73a59bf11788627de1ecb6e49f5c3e88efb39521b87dbc16658"
Oct 30 22:33:20.035131 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:20.035048    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a2115f66e34c73a59bf11788627de1ecb6e49f5c3e88efb39521b87dbc16658"} err="failed to get container status \"3a2115f66e34c73a59bf11788627de1ecb6e49f5c3e88efb39521b87dbc16658\": rpc error: code = NotFound desc = could not find container \"3a2115f66e34c73a59bf11788627de1ecb6e49f5c3e88efb39521b87dbc16658\": container with ID starting with 3a2115f66e34c73a59bf11788627de1ecb6e49f5c3e88efb39521b87dbc16658 not found: ID does not exist"
Oct 30 22:33:20.035131 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:20.035069    4473 scope.go:122] "RemoveContainer" containerID="e27c816272f81b7af973f138468330bfa3320ee4dfe39dab9e899d5518fe2f5a"
Oct 30 22:33:20.035323 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:33:20.035306    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e27c816272f81b7af973f138468330bfa3320ee4dfe39dab9e899d5518fe2f5a\": container with ID starting with e27c816272f81b7af973f138468330bfa3320ee4dfe39dab9e899d5518fe2f5a not found: ID does not exist" containerID="e27c816272f81b7af973f138468330bfa3320ee4dfe39dab9e899d5518fe2f5a"
Oct 30 22:33:20.035364 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:20.035332    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e27c816272f81b7af973f138468330bfa3320ee4dfe39dab9e899d5518fe2f5a"} err="failed to get container status \"e27c816272f81b7af973f138468330bfa3320ee4dfe39dab9e899d5518fe2f5a\": rpc error: code = NotFound desc = could not find container \"e27c816272f81b7af973f138468330bfa3320ee4dfe39dab9e899d5518fe2f5a\": container with ID starting with e27c816272f81b7af973f138468330bfa3320ee4dfe39dab9e899d5518fe2f5a not found: ID does not exist"
Oct 30 22:33:20.035364 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:20.035347    4473 scope.go:122] "RemoveContainer" containerID="108d96e7adf340e43e5f9146e3353c67952fff42d1e7aa838574209ba842be2e"
Oct 30 22:33:20.035631 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:33:20.035614    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"108d96e7adf340e43e5f9146e3353c67952fff42d1e7aa838574209ba842be2e\": container with ID starting with 108d96e7adf340e43e5f9146e3353c67952fff42d1e7aa838574209ba842be2e not found: ID does not exist" containerID="108d96e7adf340e43e5f9146e3353c67952fff42d1e7aa838574209ba842be2e"
Oct 30 22:33:20.035673 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:20.035639    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"108d96e7adf340e43e5f9146e3353c67952fff42d1e7aa838574209ba842be2e"} err="failed to get container status \"108d96e7adf340e43e5f9146e3353c67952fff42d1e7aa838574209ba842be2e\": rpc error: code = NotFound desc = could not find container \"108d96e7adf340e43e5f9146e3353c67952fff42d1e7aa838574209ba842be2e\": container with ID starting with 108d96e7adf340e43e5f9146e3353c67952fff42d1e7aa838574209ba842be2e not found: ID does not exist"
Oct 30 22:33:20.178155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:20.178115    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"]
Oct 30 22:33:20.228629 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:20.228596    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-variants-predictor-9894d954c-c4kjg"]
Oct 30 22:33:21.885871 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:21.885838    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60f47caa-b86a-4dd1-a340-37bfc5b2e36b" path="/var/lib/kubelet/pods/60f47caa-b86a-4dd1-a340-37bfc5b2e36b/volumes"
Oct 30 22:33:22.004932 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.004896    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:22.271891 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.271861    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:22.711496 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.711465    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"]
Oct 30 22:33:22.711766 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.711755    4473 cpu_manager.go:414] "containerMap: removing container" podUID="60f47caa-b86a-4dd1-a340-37bfc5b2e36b" containerName="kserve-container"
Oct 30 22:33:22.711816 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.711768    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f47caa-b86a-4dd1-a340-37bfc5b2e36b" containerName="kserve-container"
Oct 30 22:33:22.711816 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.711777    4473 cpu_manager.go:414] "containerMap: removing container" podUID="60f47caa-b86a-4dd1-a340-37bfc5b2e36b" containerName="kube-rbac-proxy"
Oct 30 22:33:22.711816 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.711782    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f47caa-b86a-4dd1-a340-37bfc5b2e36b" containerName="kube-rbac-proxy"
Oct 30 22:33:22.711816 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.711802    4473 cpu_manager.go:414] "containerMap: removing container" podUID="60f47caa-b86a-4dd1-a340-37bfc5b2e36b" containerName="storage-initializer"
Oct 30 22:33:22.711816 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.711807    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f47caa-b86a-4dd1-a340-37bfc5b2e36b" containerName="storage-initializer"
Oct 30 22:33:22.712011 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.711871    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f47caa-b86a-4dd1-a340-37bfc5b2e36b" containerName="kserve-container"
Oct 30 22:33:22.712011 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.711877    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f47caa-b86a-4dd1-a340-37bfc5b2e36b" containerName="kube-rbac-proxy"
Oct 30 22:33:22.716705 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.716686    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:22.770093 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.770060    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:33:22.770093 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.770078    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"openshift-service-ca.crt\""
Oct 30 22:33:22.770273 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.770132    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"kube-root-ca.crt\""
Oct 30 22:33:22.820801 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.820773    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"seaweedfs-s3-creds\""
Oct 30 22:33:22.820801 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.820773    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Oct 30 22:33:22.821045 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.820833    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6z45\" (UniqueName: \"kubernetes.io/projected/9d77be30-128d-469e-9908-47ec28450b54-kube-api-access-b6z45\") pod \"isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:22.821045 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.820875    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Oct 30 22:33:22.821045 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.820876    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d77be30-128d-469e-9908-47ec28450b54-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:22.821045 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.820944    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"default-dockercfg-4v4s8\""
Oct 30 22:33:22.821205 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.821037    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d77be30-128d-469e-9908-47ec28450b54-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:22.821205 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.821079    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d77be30-128d-469e-9908-47ec28450b54-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:22.821205 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.821111    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d77be30-128d-469e-9908-47ec28450b54-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:22.822446 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.822428    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"]
Oct 30 22:33:22.922441 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.922410    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6z45\" (UniqueName: \"kubernetes.io/projected/9d77be30-128d-469e-9908-47ec28450b54-kube-api-access-b6z45\") pod \"isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:22.922919 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.922447    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d77be30-128d-469e-9908-47ec28450b54-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:22.922919 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.922496    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d77be30-128d-469e-9908-47ec28450b54-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:22.922919 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.922530    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d77be30-128d-469e-9908-47ec28450b54-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:22.922919 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.922576    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d77be30-128d-469e-9908-47ec28450b54-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:22.923155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.923053    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d77be30-128d-469e-9908-47ec28450b54-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:22.923253 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.923233    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d77be30-128d-469e-9908-47ec28450b54-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:22.923343 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.923328    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d77be30-128d-469e-9908-47ec28450b54-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:22.925089 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.925071    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d77be30-128d-469e-9908-47ec28450b54-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:22.980583 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:22.980520    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6z45\" (UniqueName: \"kubernetes.io/projected/9d77be30-128d-469e-9908-47ec28450b54-kube-api-access-b6z45\") pod \"isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:23.033154 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:23.033121    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:23.454067 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:23.454043    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"]
Oct 30 22:33:23.456506 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:33:23.456471    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d77be30_128d_469e_9908_47ec28450b54.slice/crio-f7eb31fb0d8931cadb9549447b3ed0c9b8cfd4ab0885dadf672091296bcf2a54 WatchSource:0}: Error finding container f7eb31fb0d8931cadb9549447b3ed0c9b8cfd4ab0885dadf672091296bcf2a54: Status 404 returned error can't find the container with id f7eb31fb0d8931cadb9549447b3ed0c9b8cfd4ab0885dadf672091296bcf2a54
Oct 30 22:33:24.032005 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:24.031971    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq" event={"ID":"9d77be30-128d-469e-9908-47ec28450b54","Type":"ContainerStarted","Data":"18fad9af3268627aaf90dab4e99dc35abed0f01e29bf2fc497663e7202f02c7b"}
Oct 30 22:33:24.032396 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:24.032024    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq" event={"ID":"9d77be30-128d-469e-9908-47ec28450b54","Type":"ContainerStarted","Data":"f7eb31fb0d8931cadb9549447b3ed0c9b8cfd4ab0885dadf672091296bcf2a54"}
Oct 30 22:33:30.061719 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:30.061685    4473 generic.go:356] "Generic (PLEG): container finished" podID="9d77be30-128d-469e-9908-47ec28450b54" containerID="18fad9af3268627aaf90dab4e99dc35abed0f01e29bf2fc497663e7202f02c7b" exitCode=0
Oct 30 22:33:30.062145 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:30.061758    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq" event={"ID":"9d77be30-128d-469e-9908-47ec28450b54","Type":"ContainerDied","Data":"18fad9af3268627aaf90dab4e99dc35abed0f01e29bf2fc497663e7202f02c7b"}
Oct 30 22:33:31.068689 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:31.068649    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq" event={"ID":"9d77be30-128d-469e-9908-47ec28450b54","Type":"ContainerStarted","Data":"03aa6b96b849b7e69e377ab12b53d4ae4ddeee8c8f816175144b0b7e62838fd0"}
Oct 30 22:33:31.069134 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:31.068718    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq" event={"ID":"9d77be30-128d-469e-9908-47ec28450b54","Type":"ContainerStarted","Data":"e314263f4999281748901cb8f5bcf956c099b5701b10992581eb2ec326f8556e"}
Oct 30 22:33:31.069134 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:31.069064    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:31.069134 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:31.069121    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:31.239458 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:31.239392    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq" podStartSLOduration=3.2289928039999998 podStartE2EDuration="9.239380158s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.010387354s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:31.238993832 +0000 UTC m=+2429.475203174" watchObservedRunningTime="2026-10-30 22:33:31.239380158 +0000 UTC m=+2429.475589503"
Oct 30 22:33:37.078852 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:37.078817    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:33:41.304440 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:33:41.304402    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:33:46.494696 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:46.494661    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:52.986334 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:52.986266    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:53.210446 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:33:53.210413    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:05.287356 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:34:05.287323    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:11.304019 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:34:11.303980    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:34:41.303842 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:34:41.303755    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:35:05.489724 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:05.489675    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:05.914660 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:05.914630    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:35:11.306047 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:35:11.306009    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:35:16.424464 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:16.424422    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:16.923148 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:16.923107    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq" podUID="9d77be30-128d-469e-9908-47ec28450b54" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:35:18.852710 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:18.852667    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"]
Oct 30 22:35:18.853218 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:18.852993    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq" podUID="9d77be30-128d-469e-9908-47ec28450b54" containerName="kserve-container" containerID="cri-o://e314263f4999281748901cb8f5bcf956c099b5701b10992581eb2ec326f8556e" gracePeriod=30
Oct 30 22:35:18.853218 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:18.852998    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq" podUID="9d77be30-128d-469e-9908-47ec28450b54" containerName="kube-rbac-proxy" containerID="cri-o://03aa6b96b849b7e69e377ab12b53d4ae4ddeee8c8f816175144b0b7e62838fd0" gracePeriod=30
Oct 30 22:35:18.864363 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:18.864340    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:19.512465 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:19.512433    4473 generic.go:356] "Generic (PLEG): container finished" podID="9d77be30-128d-469e-9908-47ec28450b54" containerID="03aa6b96b849b7e69e377ab12b53d4ae4ddeee8c8f816175144b0b7e62838fd0" exitCode=0
Oct 30 22:35:19.512640 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:19.512473    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq" event={"ID":"9d77be30-128d-469e-9908-47ec28450b54","Type":"ContainerDied","Data":"03aa6b96b849b7e69e377ab12b53d4ae4ddeee8c8f816175144b0b7e62838fd0"}
Oct 30 22:35:20.291190 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.291161    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:35:20.399714 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.399683    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d77be30-128d-469e-9908-47ec28450b54-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d77be30-128d-469e-9908-47ec28450b54-proxy-tls\") pod \"9d77be30-128d-469e-9908-47ec28450b54\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") "
Oct 30 22:35:20.399901 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.399724    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d77be30-128d-469e-9908-47ec28450b54-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d77be30-128d-469e-9908-47ec28450b54-cabundle-cert\") pod \"9d77be30-128d-469e-9908-47ec28450b54\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") "
Oct 30 22:35:20.399901 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.399788    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d77be30-128d-469e-9908-47ec28450b54-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d77be30-128d-469e-9908-47ec28450b54-kserve-provision-location\") pod \"9d77be30-128d-469e-9908-47ec28450b54\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") "
Oct 30 22:35:20.399901 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.399843    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d77be30-128d-469e-9908-47ec28450b54-kube-api-access-b6z45\" (UniqueName: \"kubernetes.io/projected/9d77be30-128d-469e-9908-47ec28450b54-kube-api-access-b6z45\") pod \"9d77be30-128d-469e-9908-47ec28450b54\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") "
Oct 30 22:35:20.399901 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.399882    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d77be30-128d-469e-9908-47ec28450b54-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d77be30-128d-469e-9908-47ec28450b54-kube-rbac-proxy-sar-config\") pod \"9d77be30-128d-469e-9908-47ec28450b54\" (UID: \"9d77be30-128d-469e-9908-47ec28450b54\") "
Oct 30 22:35:20.400257 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.400229    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d77be30-128d-469e-9908-47ec28450b54-cabundle-cert" pod "9d77be30-128d-469e-9908-47ec28450b54" (UID: "9d77be30-128d-469e-9908-47ec28450b54"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:20.400391 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.400311    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d77be30-128d-469e-9908-47ec28450b54-kube-rbac-proxy-sar-config" pod "9d77be30-128d-469e-9908-47ec28450b54" (UID: "9d77be30-128d-469e-9908-47ec28450b54"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:20.402035 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.402006    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d77be30-128d-469e-9908-47ec28450b54-proxy-tls" pod "9d77be30-128d-469e-9908-47ec28450b54" (UID: "9d77be30-128d-469e-9908-47ec28450b54"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:20.402263 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.402239    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d77be30-128d-469e-9908-47ec28450b54-kube-api-access-b6z45" pod "9d77be30-128d-469e-9908-47ec28450b54" (UID: "9d77be30-128d-469e-9908-47ec28450b54"). InnerVolumeSpecName "kube-api-access-b6z45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:20.404680 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.404660    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d77be30-128d-469e-9908-47ec28450b54-kserve-provision-location" pod "9d77be30-128d-469e-9908-47ec28450b54" (UID: "9d77be30-128d-469e-9908-47ec28450b54"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:20.500938 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.500904    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d77be30-128d-469e-9908-47ec28450b54-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:35:20.500938 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.500932    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d77be30-128d-469e-9908-47ec28450b54-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:35:20.500938 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.500943    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d77be30-128d-469e-9908-47ec28450b54-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:35:20.501186 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.500973    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6z45\" (UniqueName: \"kubernetes.io/projected/9d77be30-128d-469e-9908-47ec28450b54-kube-api-access-b6z45\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:35:20.501186 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.500982    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d77be30-128d-469e-9908-47ec28450b54-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:35:20.518308 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.518276    4473 generic.go:356] "Generic (PLEG): container finished" podID="9d77be30-128d-469e-9908-47ec28450b54" containerID="e314263f4999281748901cb8f5bcf956c099b5701b10992581eb2ec326f8556e" exitCode=0
Oct 30 22:35:20.518450 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.518334    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"
Oct 30 22:35:20.518450 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.518343    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq" event={"ID":"9d77be30-128d-469e-9908-47ec28450b54","Type":"ContainerDied","Data":"e314263f4999281748901cb8f5bcf956c099b5701b10992581eb2ec326f8556e"}
Oct 30 22:35:20.518450 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.518393    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq" event={"ID":"9d77be30-128d-469e-9908-47ec28450b54","Type":"ContainerDied","Data":"f7eb31fb0d8931cadb9549447b3ed0c9b8cfd4ab0885dadf672091296bcf2a54"}
Oct 30 22:35:20.518450 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.518418    4473 scope.go:122] "RemoveContainer" containerID="03aa6b96b849b7e69e377ab12b53d4ae4ddeee8c8f816175144b0b7e62838fd0"
Oct 30 22:35:20.527586 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.527571    4473 scope.go:122] "RemoveContainer" containerID="e314263f4999281748901cb8f5bcf956c099b5701b10992581eb2ec326f8556e"
Oct 30 22:35:20.535044 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.535013    4473 scope.go:122] "RemoveContainer" containerID="18fad9af3268627aaf90dab4e99dc35abed0f01e29bf2fc497663e7202f02c7b"
Oct 30 22:35:20.542070 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.542049    4473 scope.go:122] "RemoveContainer" containerID="03aa6b96b849b7e69e377ab12b53d4ae4ddeee8c8f816175144b0b7e62838fd0"
Oct 30 22:35:20.542321 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:35:20.542296    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03aa6b96b849b7e69e377ab12b53d4ae4ddeee8c8f816175144b0b7e62838fd0\": container with ID starting with 03aa6b96b849b7e69e377ab12b53d4ae4ddeee8c8f816175144b0b7e62838fd0 not found: ID does not exist" containerID="03aa6b96b849b7e69e377ab12b53d4ae4ddeee8c8f816175144b0b7e62838fd0"
Oct 30 22:35:20.542367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.542337    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03aa6b96b849b7e69e377ab12b53d4ae4ddeee8c8f816175144b0b7e62838fd0"} err="failed to get container status \"03aa6b96b849b7e69e377ab12b53d4ae4ddeee8c8f816175144b0b7e62838fd0\": rpc error: code = NotFound desc = could not find container \"03aa6b96b849b7e69e377ab12b53d4ae4ddeee8c8f816175144b0b7e62838fd0\": container with ID starting with 03aa6b96b849b7e69e377ab12b53d4ae4ddeee8c8f816175144b0b7e62838fd0 not found: ID does not exist"
Oct 30 22:35:20.542367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.542356    4473 scope.go:122] "RemoveContainer" containerID="e314263f4999281748901cb8f5bcf956c099b5701b10992581eb2ec326f8556e"
Oct 30 22:35:20.542578 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:35:20.542562    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e314263f4999281748901cb8f5bcf956c099b5701b10992581eb2ec326f8556e\": container with ID starting with e314263f4999281748901cb8f5bcf956c099b5701b10992581eb2ec326f8556e not found: ID does not exist" containerID="e314263f4999281748901cb8f5bcf956c099b5701b10992581eb2ec326f8556e"
Oct 30 22:35:20.542619 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.542584    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e314263f4999281748901cb8f5bcf956c099b5701b10992581eb2ec326f8556e"} err="failed to get container status \"e314263f4999281748901cb8f5bcf956c099b5701b10992581eb2ec326f8556e\": rpc error: code = NotFound desc = could not find container \"e314263f4999281748901cb8f5bcf956c099b5701b10992581eb2ec326f8556e\": container with ID starting with e314263f4999281748901cb8f5bcf956c099b5701b10992581eb2ec326f8556e not found: ID does not exist"
Oct 30 22:35:20.542619 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.542599    4473 scope.go:122] "RemoveContainer" containerID="18fad9af3268627aaf90dab4e99dc35abed0f01e29bf2fc497663e7202f02c7b"
Oct 30 22:35:20.542820 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:35:20.542805    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18fad9af3268627aaf90dab4e99dc35abed0f01e29bf2fc497663e7202f02c7b\": container with ID starting with 18fad9af3268627aaf90dab4e99dc35abed0f01e29bf2fc497663e7202f02c7b not found: ID does not exist" containerID="18fad9af3268627aaf90dab4e99dc35abed0f01e29bf2fc497663e7202f02c7b"
Oct 30 22:35:20.542861 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.542822    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18fad9af3268627aaf90dab4e99dc35abed0f01e29bf2fc497663e7202f02c7b"} err="failed to get container status \"18fad9af3268627aaf90dab4e99dc35abed0f01e29bf2fc497663e7202f02c7b\": rpc error: code = NotFound desc = could not find container \"18fad9af3268627aaf90dab4e99dc35abed0f01e29bf2fc497663e7202f02c7b\": container with ID starting with 18fad9af3268627aaf90dab4e99dc35abed0f01e29bf2fc497663e7202f02c7b not found: ID does not exist"
Oct 30 22:35:20.687377 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.687290    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"]
Oct 30 22:35:20.690831 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:20.690807    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-v2-noslash-predictor-85c4fff8dd-qtmrq"]
Oct 30 22:35:21.886524 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:21.886492    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d77be30-128d-469e-9908-47ec28450b54" path="/var/lib/kubelet/pods/9d77be30-128d-469e-9908-47ec28450b54/volumes"
Oct 30 22:35:23.473825 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:23.473785    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:23.636640 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:23.636609    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:24.194034 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.194002    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"]
Oct 30 22:35:24.194304 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.194292    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9d77be30-128d-469e-9908-47ec28450b54" containerName="kserve-container"
Oct 30 22:35:24.194349 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.194307    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d77be30-128d-469e-9908-47ec28450b54" containerName="kserve-container"
Oct 30 22:35:24.194349 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.194320    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9d77be30-128d-469e-9908-47ec28450b54" containerName="kube-rbac-proxy"
Oct 30 22:35:24.194349 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.194325    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d77be30-128d-469e-9908-47ec28450b54" containerName="kube-rbac-proxy"
Oct 30 22:35:24.194349 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.194341    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9d77be30-128d-469e-9908-47ec28450b54" containerName="storage-initializer"
Oct 30 22:35:24.194349 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.194348    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d77be30-128d-469e-9908-47ec28450b54" containerName="storage-initializer"
Oct 30 22:35:24.194502 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.194407    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d77be30-128d-469e-9908-47ec28450b54" containerName="kube-rbac-proxy"
Oct 30 22:35:24.194502 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.194420    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d77be30-128d-469e-9908-47ec28450b54" containerName="kserve-container"
Oct 30 22:35:24.199247 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.199231    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:24.252496 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.252474    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"kube-root-ca.crt\""
Oct 30 22:35:24.252496 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.252488    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"seaweedfs-s3-creds\""
Oct 30 22:35:24.252671 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.252507    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Oct 30 22:35:24.303028 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.303002    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"openshift-service-ca.crt\""
Oct 30 22:35:24.303196 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.303111    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:35:24.303196 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.303125    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Oct 30 22:35:24.303196 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.303129    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"default-dockercfg-4v4s8\""
Oct 30 22:35:24.304850 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.304830    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"]
Oct 30 22:35:24.332077 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.332057    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:24.332162 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.332092    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n88d\" (UniqueName: \"kubernetes.io/projected/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kube-api-access-2n88d\") pod \"isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:24.332162 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.332122    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:24.332291 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.332273    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:24.332368 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.332345    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:24.433034 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.432999    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:24.433201 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.433046    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n88d\" (UniqueName: \"kubernetes.io/projected/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kube-api-access-2n88d\") pod \"isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:24.433201 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.433083    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:24.433201 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.433119    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:24.433386 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.433247    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:24.433633 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.433612    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:24.433823 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.433803    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:24.433913 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.433898    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:24.435781 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.435766    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:24.490853 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.490776    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n88d\" (UniqueName: \"kubernetes.io/projected/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kube-api-access-2n88d\") pod \"isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:24.515670 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.515640    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:24.937137 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:24.937111    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"]
Oct 30 22:35:24.939638 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:35:24.939611    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6ac03f_f15e_46b4_b1bd_8a9b3dd71007.slice/crio-010dc716a4737f714fff23527196db32de59eac05d1920079ce285372a1e7cbe WatchSource:0}: Error finding container 010dc716a4737f714fff23527196db32de59eac05d1920079ce285372a1e7cbe: Status 404 returned error can't find the container with id 010dc716a4737f714fff23527196db32de59eac05d1920079ce285372a1e7cbe
Oct 30 22:35:25.540374 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:25.540337    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n" event={"ID":"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007","Type":"ContainerStarted","Data":"dde5007d1fa0be569a6af46c07121616f4932d4a1a3527e3c9043a1d08cf5673"}
Oct 30 22:35:25.540976 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:25.540413    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n" event={"ID":"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007","Type":"ContainerStarted","Data":"010dc716a4737f714fff23527196db32de59eac05d1920079ce285372a1e7cbe"}
Oct 30 22:35:36.585071 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:36.585017    4473 generic.go:356] "Generic (PLEG): container finished" podID="9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" containerID="dde5007d1fa0be569a6af46c07121616f4932d4a1a3527e3c9043a1d08cf5673" exitCode=0
Oct 30 22:35:36.585489 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:36.585103    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n" event={"ID":"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007","Type":"ContainerDied","Data":"dde5007d1fa0be569a6af46c07121616f4932d4a1a3527e3c9043a1d08cf5673"}
Oct 30 22:35:37.592100 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:37.592063    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n" event={"ID":"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007","Type":"ContainerStarted","Data":"7c1ca02085a3afa31f4cbcd30e062a79506735ba9b548b1abfc003a5f9f9dabe"}
Oct 30 22:35:37.592479 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:37.592124    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n" event={"ID":"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007","Type":"ContainerStarted","Data":"8d70875a6309f33395f9431342607b0bb176fc80b521be0f7ee5d5eab8c5c1fd"}
Oct 30 22:35:37.592479 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:37.592318    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:37.766998 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:37.766928    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n" podStartSLOduration=2.690748864 podStartE2EDuration="13.76691436s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.076165496s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:37.766480557 +0000 UTC m=+2556.002689914" watchObservedRunningTime="2026-10-30 22:35:37.76691436 +0000 UTC m=+2556.003123705"
Oct 30 22:35:38.596789 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:38.596755    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:38.598285 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:38.598261    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n" podUID="9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 30 22:35:39.601675 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:39.601635    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n" podUID="9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 30 22:35:41.304894 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:35:41.304856    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:35:44.240162 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:44.240085    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:44.607625 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:44.607545    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:44.608179 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:44.608151    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n" podUID="9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 30 22:35:52.795829 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:52.795796    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:53.095884 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.095793    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:53.284178 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.284143    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"]
Oct 30 22:35:53.287710 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.287693    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:53.341279 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.341254    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:35:53.391770 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.391685    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:35:53.391770 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.391685    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:35:53.391770 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.391685    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:35:53.391770 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.391685    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 30 22:35:53.392018 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.391804    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:35:53.392018 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.391864    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Oct 30 22:35:53.393509 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.393487    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"]
Oct 30 22:35:53.397141 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.397118    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe21d4df-1991-4441-b181-0ac1feddb836-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:53.397255 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.397145    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe21d4df-1991-4441-b181-0ac1feddb836-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:53.397255 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.397175    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe21d4df-1991-4441-b181-0ac1feddb836-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:53.397380 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.397314    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe21d4df-1991-4441-b181-0ac1feddb836-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:53.397380 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.397343    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9bmf\" (UniqueName: \"kubernetes.io/projected/fe21d4df-1991-4441-b181-0ac1feddb836-kube-api-access-s9bmf\") pod \"isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:53.497711 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.497667    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe21d4df-1991-4441-b181-0ac1feddb836-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:53.497901 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.497735    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe21d4df-1991-4441-b181-0ac1feddb836-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:53.497901 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.497765    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9bmf\" (UniqueName: \"kubernetes.io/projected/fe21d4df-1991-4441-b181-0ac1feddb836-kube-api-access-s9bmf\") pod \"isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:53.497901 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.497806    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe21d4df-1991-4441-b181-0ac1feddb836-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:53.497901 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.497832    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe21d4df-1991-4441-b181-0ac1feddb836-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:53.498269 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.498245    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe21d4df-1991-4441-b181-0ac1feddb836-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:53.498498 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.498479    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe21d4df-1991-4441-b181-0ac1feddb836-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:53.498548 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.498481    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe21d4df-1991-4441-b181-0ac1feddb836-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:53.500369 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.500346    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe21d4df-1991-4441-b181-0ac1feddb836-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:53.557347 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.557317    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9bmf\" (UniqueName: \"kubernetes.io/projected/fe21d4df-1991-4441-b181-0ac1feddb836-kube-api-access-s9bmf\") pod \"isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:53.605422 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:53.605382    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:54.029212 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:54.029143    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"]
Oct 30 22:35:54.068029 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:35:54.067999    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe21d4df_1991_4441_b181_0ac1feddb836.slice/crio-da27810fbe8bf28b7d0362bfad8a2053429a893e43cf05b553d318ea2bfd3ff6 WatchSource:0}: Error finding container da27810fbe8bf28b7d0362bfad8a2053429a893e43cf05b553d318ea2bfd3ff6: Status 404 returned error can't find the container with id da27810fbe8bf28b7d0362bfad8a2053429a893e43cf05b553d318ea2bfd3ff6
Oct 30 22:35:54.608919 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:54.608885    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:35:54.666279 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:54.666250    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v" event={"ID":"fe21d4df-1991-4441-b181-0ac1feddb836","Type":"ContainerStarted","Data":"cb3e07218b13ed82ce11cf29b7f815b3a7f98f7108573583cb4904044d886d22"}
Oct 30 22:35:54.666448 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:54.666309    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v" event={"ID":"fe21d4df-1991-4441-b181-0ac1feddb836","Type":"ContainerStarted","Data":"da27810fbe8bf28b7d0362bfad8a2053429a893e43cf05b553d318ea2bfd3ff6"}
Oct 30 22:35:58.686259 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:58.686222    4473 generic.go:356] "Generic (PLEG): container finished" podID="fe21d4df-1991-4441-b181-0ac1feddb836" containerID="cb3e07218b13ed82ce11cf29b7f815b3a7f98f7108573583cb4904044d886d22" exitCode=0
Oct 30 22:35:58.686633 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:58.686298    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v" event={"ID":"fe21d4df-1991-4441-b181-0ac1feddb836","Type":"ContainerDied","Data":"cb3e07218b13ed82ce11cf29b7f815b3a7f98f7108573583cb4904044d886d22"}
Oct 30 22:35:59.692292 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:59.692256    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v" event={"ID":"fe21d4df-1991-4441-b181-0ac1feddb836","Type":"ContainerStarted","Data":"a63abfb6352c4a40c0c5e82175aa2298b06f6dab89c1bc93132aa78e8b1346fd"}
Oct 30 22:35:59.692684 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:59.692320    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v" event={"ID":"fe21d4df-1991-4441-b181-0ac1feddb836","Type":"ContainerStarted","Data":"8281e0e38f268468ca687a171d0afe59d7a280ce201be0ffa141de696409fa23"}
Oct 30 22:35:59.692684 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:59.692506    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:35:59.914099 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:35:59.914038    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v" podStartSLOduration=3.126794991 podStartE2EDuration="6.914023925s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.787228934s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:59.862754967 +0000 UTC m=+2578.098964319" watchObservedRunningTime="2026-10-30 22:35:59.914023925 +0000 UTC m=+2578.150233264"
Oct 30 22:36:00.696627 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:00.696591    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:36:05.817016 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:05.816971    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:06.706333 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:06.706302    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:36:11.309312 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:36:11.309272    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:36:16.510156 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:16.510122    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"]
Oct 30 22:36:16.510555 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:16.510363    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n" podUID="9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" containerName="kserve-container" containerID="cri-o://8d70875a6309f33395f9431342607b0bb176fc80b521be0f7ee5d5eab8c5c1fd" gracePeriod=30
Oct 30 22:36:16.510555 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:16.510401    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n" podUID="9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" containerName="kube-rbac-proxy" containerID="cri-o://7c1ca02085a3afa31f4cbcd30e062a79506735ba9b548b1abfc003a5f9f9dabe" gracePeriod=30
Oct 30 22:36:16.524303 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:16.524273    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:16.768758 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:16.768679    4473 generic.go:356] "Generic (PLEG): container finished" podID="9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" containerID="7c1ca02085a3afa31f4cbcd30e062a79506735ba9b548b1abfc003a5f9f9dabe" exitCode=0
Oct 30 22:36:16.768758 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:16.768739    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n" event={"ID":"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007","Type":"ContainerDied","Data":"7c1ca02085a3afa31f4cbcd30e062a79506735ba9b548b1abfc003a5f9f9dabe"}
Oct 30 22:36:18.255895 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.255869    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:36:18.415737 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.415708    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kserve-provision-location\") pod \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") "
Oct 30 22:36:18.415897 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.415775    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kube-rbac-proxy-sar-config\") pod \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") "
Oct 30 22:36:18.415897 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.415814    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-proxy-tls\") pod \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") "
Oct 30 22:36:18.415897 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.415854    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-cabundle-cert\") pod \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") "
Oct 30 22:36:18.416095 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.415904    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kube-api-access-2n88d\" (UniqueName: \"kubernetes.io/projected/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kube-api-access-2n88d\") pod \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\" (UID: \"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007\") "
Oct 30 22:36:18.416260 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.416226    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kube-rbac-proxy-sar-config" pod "9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" (UID: "9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:18.416387 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.416288    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-cabundle-cert" pod "9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" (UID: "9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:18.416785 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.416758    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kserve-provision-location" pod "9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" (UID: "9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.418159 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.418137    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-proxy-tls" pod "9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" (UID: "9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:18.418246 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.418152    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kube-api-access-2n88d" pod "9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" (UID: "9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007"). InnerVolumeSpecName "kube-api-access-2n88d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:18.517042 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.516999    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.517042 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.517035    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.517042 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.517046    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.517042 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.517055    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n88d\" (UniqueName: \"kubernetes.io/projected/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kube-api-access-2n88d\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.517320 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.517064    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.784805 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.784724    4473 generic.go:356] "Generic (PLEG): container finished" podID="9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" containerID="8d70875a6309f33395f9431342607b0bb176fc80b521be0f7ee5d5eab8c5c1fd" exitCode=0
Oct 30 22:36:18.784805 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.784781    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"
Oct 30 22:36:18.785015 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.784807    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n" event={"ID":"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007","Type":"ContainerDied","Data":"8d70875a6309f33395f9431342607b0bb176fc80b521be0f7ee5d5eab8c5c1fd"}
Oct 30 22:36:18.785015 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.784862    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n" event={"ID":"9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007","Type":"ContainerDied","Data":"010dc716a4737f714fff23527196db32de59eac05d1920079ce285372a1e7cbe"}
Oct 30 22:36:18.785015 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.784888    4473 scope.go:122] "RemoveContainer" containerID="7c1ca02085a3afa31f4cbcd30e062a79506735ba9b548b1abfc003a5f9f9dabe"
Oct 30 22:36:18.795685 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.795666    4473 scope.go:122] "RemoveContainer" containerID="8d70875a6309f33395f9431342607b0bb176fc80b521be0f7ee5d5eab8c5c1fd"
Oct 30 22:36:18.803598 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.803581    4473 scope.go:122] "RemoveContainer" containerID="dde5007d1fa0be569a6af46c07121616f4932d4a1a3527e3c9043a1d08cf5673"
Oct 30 22:36:18.810995 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.810973    4473 scope.go:122] "RemoveContainer" containerID="7c1ca02085a3afa31f4cbcd30e062a79506735ba9b548b1abfc003a5f9f9dabe"
Oct 30 22:36:18.811257 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:36:18.811234    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c1ca02085a3afa31f4cbcd30e062a79506735ba9b548b1abfc003a5f9f9dabe\": container with ID starting with 7c1ca02085a3afa31f4cbcd30e062a79506735ba9b548b1abfc003a5f9f9dabe not found: ID does not exist" containerID="7c1ca02085a3afa31f4cbcd30e062a79506735ba9b548b1abfc003a5f9f9dabe"
Oct 30 22:36:18.811335 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.811271    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c1ca02085a3afa31f4cbcd30e062a79506735ba9b548b1abfc003a5f9f9dabe"} err="failed to get container status \"7c1ca02085a3afa31f4cbcd30e062a79506735ba9b548b1abfc003a5f9f9dabe\": rpc error: code = NotFound desc = could not find container \"7c1ca02085a3afa31f4cbcd30e062a79506735ba9b548b1abfc003a5f9f9dabe\": container with ID starting with 7c1ca02085a3afa31f4cbcd30e062a79506735ba9b548b1abfc003a5f9f9dabe not found: ID does not exist"
Oct 30 22:36:18.811335 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.811298    4473 scope.go:122] "RemoveContainer" containerID="8d70875a6309f33395f9431342607b0bb176fc80b521be0f7ee5d5eab8c5c1fd"
Oct 30 22:36:18.811548 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:36:18.811528    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d70875a6309f33395f9431342607b0bb176fc80b521be0f7ee5d5eab8c5c1fd\": container with ID starting with 8d70875a6309f33395f9431342607b0bb176fc80b521be0f7ee5d5eab8c5c1fd not found: ID does not exist" containerID="8d70875a6309f33395f9431342607b0bb176fc80b521be0f7ee5d5eab8c5c1fd"
Oct 30 22:36:18.811593 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.811556    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d70875a6309f33395f9431342607b0bb176fc80b521be0f7ee5d5eab8c5c1fd"} err="failed to get container status \"8d70875a6309f33395f9431342607b0bb176fc80b521be0f7ee5d5eab8c5c1fd\": rpc error: code = NotFound desc = could not find container \"8d70875a6309f33395f9431342607b0bb176fc80b521be0f7ee5d5eab8c5c1fd\": container with ID starting with 8d70875a6309f33395f9431342607b0bb176fc80b521be0f7ee5d5eab8c5c1fd not found: ID does not exist"
Oct 30 22:36:18.811593 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.811571    4473 scope.go:122] "RemoveContainer" containerID="dde5007d1fa0be569a6af46c07121616f4932d4a1a3527e3c9043a1d08cf5673"
Oct 30 22:36:18.811820 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:36:18.811803    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde5007d1fa0be569a6af46c07121616f4932d4a1a3527e3c9043a1d08cf5673\": container with ID starting with dde5007d1fa0be569a6af46c07121616f4932d4a1a3527e3c9043a1d08cf5673 not found: ID does not exist" containerID="dde5007d1fa0be569a6af46c07121616f4932d4a1a3527e3c9043a1d08cf5673"
Oct 30 22:36:18.811875 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.811828    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde5007d1fa0be569a6af46c07121616f4932d4a1a3527e3c9043a1d08cf5673"} err="failed to get container status \"dde5007d1fa0be569a6af46c07121616f4932d4a1a3527e3c9043a1d08cf5673\": rpc error: code = NotFound desc = could not find container \"dde5007d1fa0be569a6af46c07121616f4932d4a1a3527e3c9043a1d08cf5673\": container with ID starting with dde5007d1fa0be569a6af46c07121616f4932d4a1a3527e3c9043a1d08cf5673 not found: ID does not exist"
Oct 30 22:36:18.955020 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.954980    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"]
Oct 30 22:36:18.959145 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:18.959120    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-predictor-794499ff5-lkj7n"]
Oct 30 22:36:19.886969 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:19.886929    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" path="/var/lib/kubelet/pods/9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007/volumes"
Oct 30 22:36:21.023600 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.023570    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:21.266105 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.266073    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:21.511363 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.511333    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"]
Oct 30 22:36:21.511662 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.511650    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" containerName="kube-rbac-proxy"
Oct 30 22:36:21.511728 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.511664    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" containerName="kube-rbac-proxy"
Oct 30 22:36:21.511728 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.511675    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" containerName="storage-initializer"
Oct 30 22:36:21.511728 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.511681    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" containerName="storage-initializer"
Oct 30 22:36:21.511728 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.511687    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" containerName="kserve-container"
Oct 30 22:36:21.511728 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.511693    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" containerName="kserve-container"
Oct 30 22:36:21.511880 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.511753    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" containerName="kube-rbac-proxy"
Oct 30 22:36:21.511880 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.511760    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c6ac03f-f15e-46b4-b1bd-8a9b3dd71007" containerName="kserve-container"
Oct 30 22:36:21.516444 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.516427    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:21.569778 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.569748    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"kube-root-ca.crt\""
Oct 30 22:36:21.619976 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.619918    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Oct 30 22:36:21.619976 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.619920    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"default-dockercfg-4v4s8\""
Oct 30 22:36:21.620187 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.619919    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"openshift-service-ca.crt\""
Oct 30 22:36:21.620187 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.619924    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:36:21.620187 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.620027    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"seaweedfs-s3-creds\""
Oct 30 22:36:21.620187 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.620059    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Oct 30 22:36:21.621692 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.621671    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"]
Oct 30 22:36:21.642943 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.642919    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60e8ef57-965d-4240-9c9d-8f2a301a2827-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:21.643048 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.642970    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60e8ef57-965d-4240-9c9d-8f2a301a2827-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:21.643085 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.643041    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60e8ef57-965d-4240-9c9d-8f2a301a2827-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:21.643116 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.643083    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60e8ef57-965d-4240-9c9d-8f2a301a2827-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:21.643116 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.643108    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92nct\" (UniqueName: \"kubernetes.io/projected/60e8ef57-965d-4240-9c9d-8f2a301a2827-kube-api-access-92nct\") pod \"isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:21.744001 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.743946    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60e8ef57-965d-4240-9c9d-8f2a301a2827-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:21.744001 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.744015    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92nct\" (UniqueName: \"kubernetes.io/projected/60e8ef57-965d-4240-9c9d-8f2a301a2827-kube-api-access-92nct\") pod \"isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:21.744288 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.744071    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60e8ef57-965d-4240-9c9d-8f2a301a2827-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:21.744288 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.744103    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60e8ef57-965d-4240-9c9d-8f2a301a2827-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:21.744288 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.744122    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60e8ef57-965d-4240-9c9d-8f2a301a2827-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:21.744628 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.744605    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60e8ef57-965d-4240-9c9d-8f2a301a2827-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:21.744758 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.744742    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60e8ef57-965d-4240-9c9d-8f2a301a2827-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:21.744818 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.744802    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60e8ef57-965d-4240-9c9d-8f2a301a2827-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:21.746734 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.746712    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60e8ef57-965d-4240-9c9d-8f2a301a2827-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:21.801742 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.801656    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92nct\" (UniqueName: \"kubernetes.io/projected/60e8ef57-965d-4240-9c9d-8f2a301a2827-kube-api-access-92nct\") pod \"isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") " pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:21.832897 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:21.832859    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:22.254182 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:22.254153    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"]
Oct 30 22:36:22.256593 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:36:22.256565    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e8ef57_965d_4240_9c9d_8f2a301a2827.slice/crio-2f8717cc5a9b3183e7ac18c99e1e0ad6d83b932b427b3f4576008fff086fdde8 WatchSource:0}: Error finding container 2f8717cc5a9b3183e7ac18c99e1e0ad6d83b932b427b3f4576008fff086fdde8: Status 404 returned error can't find the container with id 2f8717cc5a9b3183e7ac18c99e1e0ad6d83b932b427b3f4576008fff086fdde8
Oct 30 22:36:22.805452 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:22.805422    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh" event={"ID":"60e8ef57-965d-4240-9c9d-8f2a301a2827","Type":"ContainerStarted","Data":"5c268e8f99d48e0a7a604c7b5185f480b468d23af40f8bfa27e2ccb3f7f42e86"}
Oct 30 22:36:22.805634 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:22.805478    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh" event={"ID":"60e8ef57-965d-4240-9c9d-8f2a301a2827","Type":"ContainerStarted","Data":"2f8717cc5a9b3183e7ac18c99e1e0ad6d83b932b427b3f4576008fff086fdde8"}
Oct 30 22:36:34.856119 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:34.856081    4473 generic.go:356] "Generic (PLEG): container finished" podID="60e8ef57-965d-4240-9c9d-8f2a301a2827" containerID="5c268e8f99d48e0a7a604c7b5185f480b468d23af40f8bfa27e2ccb3f7f42e86" exitCode=0
Oct 30 22:36:34.856542 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:34.856155    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh" event={"ID":"60e8ef57-965d-4240-9c9d-8f2a301a2827","Type":"ContainerDied","Data":"5c268e8f99d48e0a7a604c7b5185f480b468d23af40f8bfa27e2ccb3f7f42e86"}
Oct 30 22:36:35.862637 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:35.862599    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh" event={"ID":"60e8ef57-965d-4240-9c9d-8f2a301a2827","Type":"ContainerStarted","Data":"018d02d36d5f596ff67559d54dbb27dc2a4e2f8e4a7b61ea3624ca0a0c00c668"}
Oct 30 22:36:35.863128 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:35.862668    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh" event={"ID":"60e8ef57-965d-4240-9c9d-8f2a301a2827","Type":"ContainerStarted","Data":"3948c39d47a6029db66cb307c30640e544cd07269976480de04cbe4fd40fc46c"}
Oct 30 22:36:35.863128 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:35.863014    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:35.863254 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:35.863165    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:35.865011 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:35.864986    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh" podUID="60e8ef57-965d-4240-9c9d-8f2a301a2827" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Oct 30 22:36:36.035277 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:36.035205    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh" podStartSLOduration=2.995390466 podStartE2EDuration="15.035183962s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.039793496s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:36.032920035 +0000 UTC m=+2614.269129394" watchObservedRunningTime="2026-10-30 22:36:36.035183962 +0000 UTC m=+2614.271393310"
Oct 30 22:36:36.710538 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:36.710511    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:36:36.867289 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:36.867245    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh" podUID="60e8ef57-965d-4240-9c9d-8f2a301a2827" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Oct 30 22:36:41.304914 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:36:41.304878    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:36:41.872139 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:41.872112    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:41.872723 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:41.872693    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh" podUID="60e8ef57-965d-4240-9c9d-8f2a301a2827" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Oct 30 22:36:43.388178 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:43.388146    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"]
Oct 30 22:36:43.388593 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:43.388370    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v" podUID="fe21d4df-1991-4441-b181-0ac1feddb836" containerName="kserve-container" containerID="cri-o://8281e0e38f268468ca687a171d0afe59d7a280ce201be0ffa141de696409fa23" gracePeriod=30
Oct 30 22:36:43.388593 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:43.388432    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v" podUID="fe21d4df-1991-4441-b181-0ac1feddb836" containerName="kube-rbac-proxy" containerID="cri-o://a63abfb6352c4a40c0c5e82175aa2298b06f6dab89c1bc93132aa78e8b1346fd" gracePeriod=30
Oct 30 22:36:43.407058 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:43.407032    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:43.898227 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:43.898191    4473 generic.go:356] "Generic (PLEG): container finished" podID="fe21d4df-1991-4441-b181-0ac1feddb836" containerID="a63abfb6352c4a40c0c5e82175aa2298b06f6dab89c1bc93132aa78e8b1346fd" exitCode=0
Oct 30 22:36:43.898397 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:43.898264    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v" event={"ID":"fe21d4df-1991-4441-b181-0ac1feddb836","Type":"ContainerDied","Data":"a63abfb6352c4a40c0c5e82175aa2298b06f6dab89c1bc93132aa78e8b1346fd"}
Oct 30 22:36:46.701363 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:46.701319    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v" podUID="fe21d4df-1991-4441-b181-0ac1feddb836" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Oct 30 22:36:46.707715 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:46.707689    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v" podUID="fe21d4df-1991-4441-b181-0ac1feddb836" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.51:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.134.0.51:8080: connect: connection refused"
Oct 30 22:36:49.831132 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.831101    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:36:49.870700 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.870670    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe21d4df-1991-4441-b181-0ac1feddb836-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe21d4df-1991-4441-b181-0ac1feddb836-cabundle-cert\") pod \"fe21d4df-1991-4441-b181-0ac1feddb836\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") "
Oct 30 22:36:49.870859 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.870752    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe21d4df-1991-4441-b181-0ac1feddb836-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe21d4df-1991-4441-b181-0ac1feddb836-kserve-provision-location\") pod \"fe21d4df-1991-4441-b181-0ac1feddb836\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") "
Oct 30 22:36:49.870859 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.870789    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe21d4df-1991-4441-b181-0ac1feddb836-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe21d4df-1991-4441-b181-0ac1feddb836-kube-rbac-proxy-sar-config\") pod \"fe21d4df-1991-4441-b181-0ac1feddb836\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") "
Oct 30 22:36:49.870859 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.870844    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe21d4df-1991-4441-b181-0ac1feddb836-kube-api-access-s9bmf\" (UniqueName: \"kubernetes.io/projected/fe21d4df-1991-4441-b181-0ac1feddb836-kube-api-access-s9bmf\") pod \"fe21d4df-1991-4441-b181-0ac1feddb836\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") "
Oct 30 22:36:49.871062 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.870873    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe21d4df-1991-4441-b181-0ac1feddb836-proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe21d4df-1991-4441-b181-0ac1feddb836-proxy-tls\") pod \"fe21d4df-1991-4441-b181-0ac1feddb836\" (UID: \"fe21d4df-1991-4441-b181-0ac1feddb836\") "
Oct 30 22:36:49.871164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.871079    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe21d4df-1991-4441-b181-0ac1feddb836-cabundle-cert" pod "fe21d4df-1991-4441-b181-0ac1feddb836" (UID: "fe21d4df-1991-4441-b181-0ac1feddb836"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:49.871283 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.871142    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe21d4df-1991-4441-b181-0ac1feddb836-kserve-provision-location" pod "fe21d4df-1991-4441-b181-0ac1feddb836" (UID: "fe21d4df-1991-4441-b181-0ac1feddb836"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:49.871283 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.871206    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe21d4df-1991-4441-b181-0ac1feddb836-kube-rbac-proxy-sar-config" pod "fe21d4df-1991-4441-b181-0ac1feddb836" (UID: "fe21d4df-1991-4441-b181-0ac1feddb836"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:49.873151 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.873126    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe21d4df-1991-4441-b181-0ac1feddb836-proxy-tls" pod "fe21d4df-1991-4441-b181-0ac1feddb836" (UID: "fe21d4df-1991-4441-b181-0ac1feddb836"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:49.873151 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.873132    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe21d4df-1991-4441-b181-0ac1feddb836-kube-api-access-s9bmf" pod "fe21d4df-1991-4441-b181-0ac1feddb836" (UID: "fe21d4df-1991-4441-b181-0ac1feddb836"). InnerVolumeSpecName "kube-api-access-s9bmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:49.926647 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.926574    4473 generic.go:356] "Generic (PLEG): container finished" podID="fe21d4df-1991-4441-b181-0ac1feddb836" containerID="8281e0e38f268468ca687a171d0afe59d7a280ce201be0ffa141de696409fa23" exitCode=0
Oct 30 22:36:49.926647 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.926635    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v" event={"ID":"fe21d4df-1991-4441-b181-0ac1feddb836","Type":"ContainerDied","Data":"8281e0e38f268468ca687a171d0afe59d7a280ce201be0ffa141de696409fa23"}
Oct 30 22:36:49.926647 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.926645    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"
Oct 30 22:36:49.926836 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.926679    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v" event={"ID":"fe21d4df-1991-4441-b181-0ac1feddb836","Type":"ContainerDied","Data":"da27810fbe8bf28b7d0362bfad8a2053429a893e43cf05b553d318ea2bfd3ff6"}
Oct 30 22:36:49.926836 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.926704    4473 scope.go:122] "RemoveContainer" containerID="a63abfb6352c4a40c0c5e82175aa2298b06f6dab89c1bc93132aa78e8b1346fd"
Oct 30 22:36:49.934932 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.934914    4473 scope.go:122] "RemoveContainer" containerID="8281e0e38f268468ca687a171d0afe59d7a280ce201be0ffa141de696409fa23"
Oct 30 22:36:49.942180 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.942161    4473 scope.go:122] "RemoveContainer" containerID="cb3e07218b13ed82ce11cf29b7f815b3a7f98f7108573583cb4904044d886d22"
Oct 30 22:36:49.949379 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.949363    4473 scope.go:122] "RemoveContainer" containerID="a63abfb6352c4a40c0c5e82175aa2298b06f6dab89c1bc93132aa78e8b1346fd"
Oct 30 22:36:49.949623 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:36:49.949604    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a63abfb6352c4a40c0c5e82175aa2298b06f6dab89c1bc93132aa78e8b1346fd\": container with ID starting with a63abfb6352c4a40c0c5e82175aa2298b06f6dab89c1bc93132aa78e8b1346fd not found: ID does not exist" containerID="a63abfb6352c4a40c0c5e82175aa2298b06f6dab89c1bc93132aa78e8b1346fd"
Oct 30 22:36:49.949685 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.949631    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63abfb6352c4a40c0c5e82175aa2298b06f6dab89c1bc93132aa78e8b1346fd"} err="failed to get container status \"a63abfb6352c4a40c0c5e82175aa2298b06f6dab89c1bc93132aa78e8b1346fd\": rpc error: code = NotFound desc = could not find container \"a63abfb6352c4a40c0c5e82175aa2298b06f6dab89c1bc93132aa78e8b1346fd\": container with ID starting with a63abfb6352c4a40c0c5e82175aa2298b06f6dab89c1bc93132aa78e8b1346fd not found: ID does not exist"
Oct 30 22:36:49.949685 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.949648    4473 scope.go:122] "RemoveContainer" containerID="8281e0e38f268468ca687a171d0afe59d7a280ce201be0ffa141de696409fa23"
Oct 30 22:36:49.949889 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:36:49.949874    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8281e0e38f268468ca687a171d0afe59d7a280ce201be0ffa141de696409fa23\": container with ID starting with 8281e0e38f268468ca687a171d0afe59d7a280ce201be0ffa141de696409fa23 not found: ID does not exist" containerID="8281e0e38f268468ca687a171d0afe59d7a280ce201be0ffa141de696409fa23"
Oct 30 22:36:49.949929 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.949892    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8281e0e38f268468ca687a171d0afe59d7a280ce201be0ffa141de696409fa23"} err="failed to get container status \"8281e0e38f268468ca687a171d0afe59d7a280ce201be0ffa141de696409fa23\": rpc error: code = NotFound desc = could not find container \"8281e0e38f268468ca687a171d0afe59d7a280ce201be0ffa141de696409fa23\": container with ID starting with 8281e0e38f268468ca687a171d0afe59d7a280ce201be0ffa141de696409fa23 not found: ID does not exist"
Oct 30 22:36:49.949929 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.949904    4473 scope.go:122] "RemoveContainer" containerID="cb3e07218b13ed82ce11cf29b7f815b3a7f98f7108573583cb4904044d886d22"
Oct 30 22:36:49.950138 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:36:49.950123    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb3e07218b13ed82ce11cf29b7f815b3a7f98f7108573583cb4904044d886d22\": container with ID starting with cb3e07218b13ed82ce11cf29b7f815b3a7f98f7108573583cb4904044d886d22 not found: ID does not exist" containerID="cb3e07218b13ed82ce11cf29b7f815b3a7f98f7108573583cb4904044d886d22"
Oct 30 22:36:49.950184 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.950142    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb3e07218b13ed82ce11cf29b7f815b3a7f98f7108573583cb4904044d886d22"} err="failed to get container status \"cb3e07218b13ed82ce11cf29b7f815b3a7f98f7108573583cb4904044d886d22\": rpc error: code = NotFound desc = could not find container \"cb3e07218b13ed82ce11cf29b7f815b3a7f98f7108573583cb4904044d886d22\": container with ID starting with cb3e07218b13ed82ce11cf29b7f815b3a7f98f7108573583cb4904044d886d22 not found: ID does not exist"
Oct 30 22:36:49.971663 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.971640    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9bmf\" (UniqueName: \"kubernetes.io/projected/fe21d4df-1991-4441-b181-0ac1feddb836-kube-api-access-s9bmf\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:49.971729 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.971664    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe21d4df-1991-4441-b181-0ac1feddb836-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:49.971729 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.971674    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe21d4df-1991-4441-b181-0ac1feddb836-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:49.971729 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.971687    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe21d4df-1991-4441-b181-0ac1feddb836-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:49.971729 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:49.971700    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe21d4df-1991-4441-b181-0ac1feddb836-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:50.117672 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:50.117638    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"]
Oct 30 22:36:50.121382 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:50.121355    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-mlserver-predictor-6ff56c85bc-wgz9v"]
Oct 30 22:36:51.873114 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:51.873090    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:36:51.886318 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:51.886294    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe21d4df-1991-4441-b181-0ac1feddb836" path="/var/lib/kubelet/pods/fe21d4df-1991-4441-b181-0ac1feddb836/volumes"
Oct 30 22:36:51.914488 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:51.914428    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:52.082117 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.082087    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:52.402158 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.402116    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"]
Oct 30 22:36:52.402460 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.402447    4473 cpu_manager.go:414] "containerMap: removing container" podUID="fe21d4df-1991-4441-b181-0ac1feddb836" containerName="kube-rbac-proxy"
Oct 30 22:36:52.402504 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.402462    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe21d4df-1991-4441-b181-0ac1feddb836" containerName="kube-rbac-proxy"
Oct 30 22:36:52.402504 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.402475    4473 cpu_manager.go:414] "containerMap: removing container" podUID="fe21d4df-1991-4441-b181-0ac1feddb836" containerName="kserve-container"
Oct 30 22:36:52.402504 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.402480    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe21d4df-1991-4441-b181-0ac1feddb836" containerName="kserve-container"
Oct 30 22:36:52.402594 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.402505    4473 cpu_manager.go:414] "containerMap: removing container" podUID="fe21d4df-1991-4441-b181-0ac1feddb836" containerName="storage-initializer"
Oct 30 22:36:52.402594 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.402510    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe21d4df-1991-4441-b181-0ac1feddb836" containerName="storage-initializer"
Oct 30 22:36:52.402594 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.402574    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe21d4df-1991-4441-b181-0ac1feddb836" containerName="kube-rbac-proxy"
Oct 30 22:36:52.402594 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.402582    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe21d4df-1991-4441-b181-0ac1feddb836" containerName="kserve-container"
Oct 30 22:36:52.407225 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.407205    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:52.460430 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.460399    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:36:52.494518 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.494490    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-78bcc674db-gkk8h\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") " pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:52.494698 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.494525    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp9g7\" (UniqueName: \"kubernetes.io/projected/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kube-api-access-lp9g7\") pod \"xgboost-v2-mlserver-predictor-78bcc674db-gkk8h\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") " pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:52.494698 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.494557    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48a8c8f1-3b5a-4515-8327-9acea448c6d3-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-78bcc674db-gkk8h\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") " pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:52.494698 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.494663    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48a8c8f1-3b5a-4515-8327-9acea448c6d3-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-78bcc674db-gkk8h\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") " pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:52.494833 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.494704    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-78bcc674db-gkk8h\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") " pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:52.510804 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.510776    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:36:52.510804 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.510774    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:36:52.510804 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.510772    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 30 22:36:52.511133 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.510775    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Oct 30 22:36:52.511133 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.510936    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:36:52.511133 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.511006    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:36:52.512552 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.512530    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"]
Oct 30 22:36:52.595085 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.595054    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48a8c8f1-3b5a-4515-8327-9acea448c6d3-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-78bcc674db-gkk8h\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") " pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:52.595085 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.595096    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-78bcc674db-gkk8h\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") " pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:52.595363 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.595132    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-78bcc674db-gkk8h\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") " pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:52.595363 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.595166    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp9g7\" (UniqueName: \"kubernetes.io/projected/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kube-api-access-lp9g7\") pod \"xgboost-v2-mlserver-predictor-78bcc674db-gkk8h\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") " pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:52.595363 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.595218    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48a8c8f1-3b5a-4515-8327-9acea448c6d3-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-78bcc674db-gkk8h\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") " pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:52.595571 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.595543    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-78bcc674db-gkk8h\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") " pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:52.596243 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.596215    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-78bcc674db-gkk8h\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") " pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:52.596348 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.596215    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48a8c8f1-3b5a-4515-8327-9acea448c6d3-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-78bcc674db-gkk8h\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") " pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:52.598051 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.598032    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48a8c8f1-3b5a-4515-8327-9acea448c6d3-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-78bcc674db-gkk8h\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") " pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:52.653260 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.653182    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp9g7\" (UniqueName: \"kubernetes.io/projected/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kube-api-access-lp9g7\") pod \"xgboost-v2-mlserver-predictor-78bcc674db-gkk8h\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") " pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:52.723900 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:52.723868    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:53.153551 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:53.153529    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"]
Oct 30 22:36:53.156120 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:36:53.156092    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a8c8f1_3b5a_4515_8327_9acea448c6d3.slice/crio-931133140dc6dc1a865c306709a10200fbf719527aad35a2570b0fbd8ced87ae WatchSource:0}: Error finding container 931133140dc6dc1a865c306709a10200fbf719527aad35a2570b0fbd8ced87ae: Status 404 returned error can't find the container with id 931133140dc6dc1a865c306709a10200fbf719527aad35a2570b0fbd8ced87ae
Oct 30 22:36:53.945577 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:53.945539    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h" event={"ID":"48a8c8f1-3b5a-4515-8327-9acea448c6d3","Type":"ContainerStarted","Data":"203e34c62ae74c8c0e6ab8ead98b0a87963e4818081bafe2710b01e45b9b823a"}
Oct 30 22:36:53.945759 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:53.945598    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h" event={"ID":"48a8c8f1-3b5a-4515-8327-9acea448c6d3","Type":"ContainerStarted","Data":"931133140dc6dc1a865c306709a10200fbf719527aad35a2570b0fbd8ced87ae"}
Oct 30 22:36:57.963611 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:57.963579    4473 generic.go:356] "Generic (PLEG): container finished" podID="48a8c8f1-3b5a-4515-8327-9acea448c6d3" containerID="203e34c62ae74c8c0e6ab8ead98b0a87963e4818081bafe2710b01e45b9b823a" exitCode=0
Oct 30 22:36:57.964031 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:57.963625    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h" event={"ID":"48a8c8f1-3b5a-4515-8327-9acea448c6d3","Type":"ContainerDied","Data":"203e34c62ae74c8c0e6ab8ead98b0a87963e4818081bafe2710b01e45b9b823a"}
Oct 30 22:36:58.969917 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:58.969874    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h" event={"ID":"48a8c8f1-3b5a-4515-8327-9acea448c6d3","Type":"ContainerStarted","Data":"0ccbc5d04161b4c30c5a3a6261df3bcadc22a1ea27b4edcefa6c55f033ac1af4"}
Oct 30 22:36:58.970466 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:58.969938    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h" event={"ID":"48a8c8f1-3b5a-4515-8327-9acea448c6d3","Type":"ContainerStarted","Data":"13111bd285051b16372d97e7d1f36fdbcb905f4f1f2cfa555ccdfa519d5f7e24"}
Oct 30 22:36:58.970466 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:58.970137    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:36:59.091581 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:59.091523    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h" podStartSLOduration=3.068201673 podStartE2EDuration="7.091509017s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.023307344s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:59.091068947 +0000 UTC m=+2637.327278295" watchObservedRunningTime="2026-10-30 22:36:59.091509017 +0000 UTC m=+2637.327718356"
Oct 30 22:36:59.737635 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:59.737604    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:59.974448 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:36:59.974415    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:37:05.983742 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:05.983708    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:37:06.143614 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:06.143583    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:06.873747 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:06.873712    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"]
Oct 30 22:37:06.874021 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:06.873998    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh" podUID="60e8ef57-965d-4240-9c9d-8f2a301a2827" containerName="kserve-container" containerID="cri-o://3948c39d47a6029db66cb307c30640e544cd07269976480de04cbe4fd40fc46c" gracePeriod=30
Oct 30 22:37:06.874106 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:06.874024    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh" podUID="60e8ef57-965d-4240-9c9d-8f2a301a2827" containerName="kube-rbac-proxy" containerID="cri-o://018d02d36d5f596ff67559d54dbb27dc2a4e2f8e4a7b61ea3624ca0a0c00c668" gracePeriod=30
Oct 30 22:37:06.877772 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:06.877746    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:07.005567 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:07.005537    4473 generic.go:356] "Generic (PLEG): container finished" podID="60e8ef57-965d-4240-9c9d-8f2a301a2827" containerID="018d02d36d5f596ff67559d54dbb27dc2a4e2f8e4a7b61ea3624ca0a0c00c668" exitCode=0
Oct 30 22:37:07.005939 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:07.005576    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh" event={"ID":"60e8ef57-965d-4240-9c9d-8f2a301a2827","Type":"ContainerDied","Data":"018d02d36d5f596ff67559d54dbb27dc2a4e2f8e4a7b61ea3624ca0a0c00c668"}
Oct 30 22:37:08.316133 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:08.316106    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:37:08.425390 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:08.425346    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e8ef57-965d-4240-9c9d-8f2a301a2827-proxy-tls\" (UniqueName: \"kubernetes.io/secret/60e8ef57-965d-4240-9c9d-8f2a301a2827-proxy-tls\") pod \"60e8ef57-965d-4240-9c9d-8f2a301a2827\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") "
Oct 30 22:37:08.425602 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:08.425438    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e8ef57-965d-4240-9c9d-8f2a301a2827-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60e8ef57-965d-4240-9c9d-8f2a301a2827-kube-rbac-proxy-sar-config\") pod \"60e8ef57-965d-4240-9c9d-8f2a301a2827\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") "
Oct 30 22:37:08.425602 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:08.425503    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e8ef57-965d-4240-9c9d-8f2a301a2827-kube-api-access-92nct\" (UniqueName: \"kubernetes.io/projected/60e8ef57-965d-4240-9c9d-8f2a301a2827-kube-api-access-92nct\") pod \"60e8ef57-965d-4240-9c9d-8f2a301a2827\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") "
Oct 30 22:37:08.425602 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:08.425538    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e8ef57-965d-4240-9c9d-8f2a301a2827-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60e8ef57-965d-4240-9c9d-8f2a301a2827-kserve-provision-location\") pod \"60e8ef57-965d-4240-9c9d-8f2a301a2827\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") "
Oct 30 22:37:08.425602 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:08.425556    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e8ef57-965d-4240-9c9d-8f2a301a2827-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60e8ef57-965d-4240-9c9d-8f2a301a2827-cabundle-cert\") pod \"60e8ef57-965d-4240-9c9d-8f2a301a2827\" (UID: \"60e8ef57-965d-4240-9c9d-8f2a301a2827\") "
Oct 30 22:37:08.425968 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:08.425917    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e8ef57-965d-4240-9c9d-8f2a301a2827-kube-rbac-proxy-sar-config" pod "60e8ef57-965d-4240-9c9d-8f2a301a2827" (UID: "60e8ef57-965d-4240-9c9d-8f2a301a2827"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:08.425968 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:08.425928    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e8ef57-965d-4240-9c9d-8f2a301a2827-cabundle-cert" pod "60e8ef57-965d-4240-9c9d-8f2a301a2827" (UID: "60e8ef57-965d-4240-9c9d-8f2a301a2827"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:08.426578 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:08.426560    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e8ef57-965d-4240-9c9d-8f2a301a2827-kserve-provision-location" pod "60e8ef57-965d-4240-9c9d-8f2a301a2827" (UID: "60e8ef57-965d-4240-9c9d-8f2a301a2827"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:08.427683 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:08.427662    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e8ef57-965d-4240-9c9d-8f2a301a2827-proxy-tls" pod "60e8ef57-965d-4240-9c9d-8f2a301a2827" (UID: "60e8ef57-965d-4240-9c9d-8f2a301a2827"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:37:08.427866 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:08.427844    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e8ef57-965d-4240-9c9d-8f2a301a2827-kube-api-access-92nct" pod "60e8ef57-965d-4240-9c9d-8f2a301a2827" (UID: "60e8ef57-965d-4240-9c9d-8f2a301a2827"). InnerVolumeSpecName "kube-api-access-92nct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:08.526488 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:08.526437    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92nct\" (UniqueName: \"kubernetes.io/projected/60e8ef57-965d-4240-9c9d-8f2a301a2827-kube-api-access-92nct\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:08.526488 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:08.526484    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60e8ef57-965d-4240-9c9d-8f2a301a2827-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:08.526488 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:08.526501    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60e8ef57-965d-4240-9c9d-8f2a301a2827-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:08.526731 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:08.526514    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60e8ef57-965d-4240-9c9d-8f2a301a2827-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:08.526731 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:08.526527    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60e8ef57-965d-4240-9c9d-8f2a301a2827-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:09.015540 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:09.015504    4473 generic.go:356] "Generic (PLEG): container finished" podID="60e8ef57-965d-4240-9c9d-8f2a301a2827" containerID="3948c39d47a6029db66cb307c30640e544cd07269976480de04cbe4fd40fc46c" exitCode=0
Oct 30 22:37:09.015710 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:09.015562    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"
Oct 30 22:37:09.015710 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:09.015566    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh" event={"ID":"60e8ef57-965d-4240-9c9d-8f2a301a2827","Type":"ContainerDied","Data":"3948c39d47a6029db66cb307c30640e544cd07269976480de04cbe4fd40fc46c"}
Oct 30 22:37:09.015710 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:09.015679    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh" event={"ID":"60e8ef57-965d-4240-9c9d-8f2a301a2827","Type":"ContainerDied","Data":"2f8717cc5a9b3183e7ac18c99e1e0ad6d83b932b427b3f4576008fff086fdde8"}
Oct 30 22:37:09.015710 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:09.015706    4473 scope.go:122] "RemoveContainer" containerID="018d02d36d5f596ff67559d54dbb27dc2a4e2f8e4a7b61ea3624ca0a0c00c668"
Oct 30 22:37:09.024358 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:09.024328    4473 scope.go:122] "RemoveContainer" containerID="3948c39d47a6029db66cb307c30640e544cd07269976480de04cbe4fd40fc46c"
Oct 30 22:37:09.031996 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:09.031978    4473 scope.go:122] "RemoveContainer" containerID="5c268e8f99d48e0a7a604c7b5185f480b468d23af40f8bfa27e2ccb3f7f42e86"
Oct 30 22:37:09.039209 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:09.039191    4473 scope.go:122] "RemoveContainer" containerID="018d02d36d5f596ff67559d54dbb27dc2a4e2f8e4a7b61ea3624ca0a0c00c668"
Oct 30 22:37:09.039475 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:37:09.039457    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018d02d36d5f596ff67559d54dbb27dc2a4e2f8e4a7b61ea3624ca0a0c00c668\": container with ID starting with 018d02d36d5f596ff67559d54dbb27dc2a4e2f8e4a7b61ea3624ca0a0c00c668 not found: ID does not exist" containerID="018d02d36d5f596ff67559d54dbb27dc2a4e2f8e4a7b61ea3624ca0a0c00c668"
Oct 30 22:37:09.039527 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:09.039485    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018d02d36d5f596ff67559d54dbb27dc2a4e2f8e4a7b61ea3624ca0a0c00c668"} err="failed to get container status \"018d02d36d5f596ff67559d54dbb27dc2a4e2f8e4a7b61ea3624ca0a0c00c668\": rpc error: code = NotFound desc = could not find container \"018d02d36d5f596ff67559d54dbb27dc2a4e2f8e4a7b61ea3624ca0a0c00c668\": container with ID starting with 018d02d36d5f596ff67559d54dbb27dc2a4e2f8e4a7b61ea3624ca0a0c00c668 not found: ID does not exist"
Oct 30 22:37:09.039527 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:09.039503    4473 scope.go:122] "RemoveContainer" containerID="3948c39d47a6029db66cb307c30640e544cd07269976480de04cbe4fd40fc46c"
Oct 30 22:37:09.039752 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:37:09.039735    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3948c39d47a6029db66cb307c30640e544cd07269976480de04cbe4fd40fc46c\": container with ID starting with 3948c39d47a6029db66cb307c30640e544cd07269976480de04cbe4fd40fc46c not found: ID does not exist" containerID="3948c39d47a6029db66cb307c30640e544cd07269976480de04cbe4fd40fc46c"
Oct 30 22:37:09.039796 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:09.039759    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3948c39d47a6029db66cb307c30640e544cd07269976480de04cbe4fd40fc46c"} err="failed to get container status \"3948c39d47a6029db66cb307c30640e544cd07269976480de04cbe4fd40fc46c\": rpc error: code = NotFound desc = could not find container \"3948c39d47a6029db66cb307c30640e544cd07269976480de04cbe4fd40fc46c\": container with ID starting with 3948c39d47a6029db66cb307c30640e544cd07269976480de04cbe4fd40fc46c not found: ID does not exist"
Oct 30 22:37:09.039796 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:09.039775    4473 scope.go:122] "RemoveContainer" containerID="5c268e8f99d48e0a7a604c7b5185f480b468d23af40f8bfa27e2ccb3f7f42e86"
Oct 30 22:37:09.040018 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:37:09.040000    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c268e8f99d48e0a7a604c7b5185f480b468d23af40f8bfa27e2ccb3f7f42e86\": container with ID starting with 5c268e8f99d48e0a7a604c7b5185f480b468d23af40f8bfa27e2ccb3f7f42e86 not found: ID does not exist" containerID="5c268e8f99d48e0a7a604c7b5185f480b468d23af40f8bfa27e2ccb3f7f42e86"
Oct 30 22:37:09.040074 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:09.040021    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c268e8f99d48e0a7a604c7b5185f480b468d23af40f8bfa27e2ccb3f7f42e86"} err="failed to get container status \"5c268e8f99d48e0a7a604c7b5185f480b468d23af40f8bfa27e2ccb3f7f42e86\": rpc error: code = NotFound desc = could not find container \"5c268e8f99d48e0a7a604c7b5185f480b468d23af40f8bfa27e2ccb3f7f42e86\": container with ID starting with 5c268e8f99d48e0a7a604c7b5185f480b468d23af40f8bfa27e2ccb3f7f42e86 not found: ID does not exist"
Oct 30 22:37:09.232446 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:09.232411    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"]
Oct 30 22:37:09.283166 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:09.283083    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-autogluon-ts-v1-noslash-predictor-6569b4f9f6-tvtxh"]
Oct 30 22:37:09.886847 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:09.886815    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e8ef57-965d-4240-9c9d-8f2a301a2827" path="/var/lib/kubelet/pods/60e8ef57-965d-4240-9c9d-8f2a301a2827/volumes"
Oct 30 22:37:11.303593 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:37:11.303557    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:37:12.513059 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:12.512983    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:12.852804 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:12.852715    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:13.003684 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.003654    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"]
Oct 30 22:37:13.004165 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.004150    4473 cpu_manager.go:414] "containerMap: removing container" podUID="60e8ef57-965d-4240-9c9d-8f2a301a2827" containerName="storage-initializer"
Oct 30 22:37:13.004249 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.004168    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e8ef57-965d-4240-9c9d-8f2a301a2827" containerName="storage-initializer"
Oct 30 22:37:13.004249 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.004181    4473 cpu_manager.go:414] "containerMap: removing container" podUID="60e8ef57-965d-4240-9c9d-8f2a301a2827" containerName="kserve-container"
Oct 30 22:37:13.004249 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.004190    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e8ef57-965d-4240-9c9d-8f2a301a2827" containerName="kserve-container"
Oct 30 22:37:13.004249 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.004216    4473 cpu_manager.go:414] "containerMap: removing container" podUID="60e8ef57-965d-4240-9c9d-8f2a301a2827" containerName="kube-rbac-proxy"
Oct 30 22:37:13.004249 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.004224    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e8ef57-965d-4240-9c9d-8f2a301a2827" containerName="kube-rbac-proxy"
Oct 30 22:37:13.004464 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.004290    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e8ef57-965d-4240-9c9d-8f2a301a2827" containerName="kserve-container"
Oct 30 22:37:13.004464 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.004305    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e8ef57-965d-4240-9c9d-8f2a301a2827" containerName="kube-rbac-proxy"
Oct 30 22:37:13.009426 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.009406    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:13.042770 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.042729    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:13.062924 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.062901    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"seaweedfs-s3-creds\""
Oct 30 22:37:13.062924 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.062925    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:37:13.113344 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.113256    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"default-dockercfg-4v4s8\""
Oct 30 22:37:13.113344 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.113269    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Oct 30 22:37:13.113581 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.113408    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"kube-root-ca.crt\""
Oct 30 22:37:13.113581 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.113466    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"openshift-service-ca.crt\""
Oct 30 22:37:13.113581 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.113491    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Oct 30 22:37:13.115069 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.115045    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"]
Oct 30 22:37:13.161367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.161336    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1747489c-7deb-499b-a98d-4b86fa70a15a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:13.161528 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.161399    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1747489c-7deb-499b-a98d-4b86fa70a15a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:13.161528 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.161484    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1747489c-7deb-499b-a98d-4b86fa70a15a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:13.161628 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.161547    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1747489c-7deb-499b-a98d-4b86fa70a15a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:13.161628 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.161580    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf98x\" (UniqueName: \"kubernetes.io/projected/1747489c-7deb-499b-a98d-4b86fa70a15a-kube-api-access-bf98x\") pod \"isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:13.233229 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.233182    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:13.262636 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.262605    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1747489c-7deb-499b-a98d-4b86fa70a15a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:13.262636 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.262637    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1747489c-7deb-499b-a98d-4b86fa70a15a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:13.262876 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.262668    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf98x\" (UniqueName: \"kubernetes.io/projected/1747489c-7deb-499b-a98d-4b86fa70a15a-kube-api-access-bf98x\") pod \"isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:13.262876 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.262689    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1747489c-7deb-499b-a98d-4b86fa70a15a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:13.262876 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.262741    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1747489c-7deb-499b-a98d-4b86fa70a15a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:13.263244 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.263219    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1747489c-7deb-499b-a98d-4b86fa70a15a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:13.263434 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.263411    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1747489c-7deb-499b-a98d-4b86fa70a15a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:13.263499 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.263445    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1747489c-7deb-499b-a98d-4b86fa70a15a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:13.265419 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.265396    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1747489c-7deb-499b-a98d-4b86fa70a15a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:13.320512 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.320474    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf98x\" (UniqueName: \"kubernetes.io/projected/1747489c-7deb-499b-a98d-4b86fa70a15a-kube-api-access-bf98x\") pod \"isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:13.326428 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.326406    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:13.536022 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.535986    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"]
Oct 30 22:37:13.539784 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.539769    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:13.592530 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.592495    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-canary-create-predictor-serving-cert\""
Oct 30 22:37:13.599635 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.599611    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"]
Oct 30 22:37:13.666465 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.666432    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3304e79c-2498-4fc7-b4f4-093f404783f5-proxy-tls\") pod \"isvc-canary-create-predictor-86d48b7dcf-fvqfw\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:13.666642 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.666469    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3304e79c-2498-4fc7-b4f4-093f404783f5-cabundle-cert\") pod \"isvc-canary-create-predictor-86d48b7dcf-fvqfw\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:13.666642 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.666492    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj4dw\" (UniqueName: \"kubernetes.io/projected/3304e79c-2498-4fc7-b4f4-093f404783f5-kube-api-access-pj4dw\") pod \"isvc-canary-create-predictor-86d48b7dcf-fvqfw\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:13.666730 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.666656    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3304e79c-2498-4fc7-b4f4-093f404783f5-kserve-provision-location\") pod \"isvc-canary-create-predictor-86d48b7dcf-fvqfw\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:13.666806 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.666787    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3304e79c-2498-4fc7-b4f4-093f404783f5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-86d48b7dcf-fvqfw\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:13.728303 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.728282    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"]
Oct 30 22:37:13.730924 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:37:13.730898    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1747489c_7deb_499b_a98d_4b86fa70a15a.slice/crio-66f1b69308d1c1500aa745a05cc8ccc0e7adc05ce132e06f4c0a1d95eb2fade6 WatchSource:0}: Error finding container 66f1b69308d1c1500aa745a05cc8ccc0e7adc05ce132e06f4c0a1d95eb2fade6: Status 404 returned error can't find the container with id 66f1b69308d1c1500aa745a05cc8ccc0e7adc05ce132e06f4c0a1d95eb2fade6
Oct 30 22:37:13.767914 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.767886    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3304e79c-2498-4fc7-b4f4-093f404783f5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-86d48b7dcf-fvqfw\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:13.768075 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.767928    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3304e79c-2498-4fc7-b4f4-093f404783f5-proxy-tls\") pod \"isvc-canary-create-predictor-86d48b7dcf-fvqfw\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:13.768075 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.767990    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3304e79c-2498-4fc7-b4f4-093f404783f5-cabundle-cert\") pod \"isvc-canary-create-predictor-86d48b7dcf-fvqfw\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:13.768075 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.768020    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj4dw\" (UniqueName: \"kubernetes.io/projected/3304e79c-2498-4fc7-b4f4-093f404783f5-kube-api-access-pj4dw\") pod \"isvc-canary-create-predictor-86d48b7dcf-fvqfw\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:13.768075 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.768065    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3304e79c-2498-4fc7-b4f4-093f404783f5-kserve-provision-location\") pod \"isvc-canary-create-predictor-86d48b7dcf-fvqfw\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:13.768452 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.768431    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3304e79c-2498-4fc7-b4f4-093f404783f5-kserve-provision-location\") pod \"isvc-canary-create-predictor-86d48b7dcf-fvqfw\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:13.768720 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.768698    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3304e79c-2498-4fc7-b4f4-093f404783f5-cabundle-cert\") pod \"isvc-canary-create-predictor-86d48b7dcf-fvqfw\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:13.768776 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.768698    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3304e79c-2498-4fc7-b4f4-093f404783f5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-86d48b7dcf-fvqfw\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:13.770674 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.770653    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3304e79c-2498-4fc7-b4f4-093f404783f5-proxy-tls\") pod \"isvc-canary-create-predictor-86d48b7dcf-fvqfw\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:13.826282 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.826249    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj4dw\" (UniqueName: \"kubernetes.io/projected/3304e79c-2498-4fc7-b4f4-093f404783f5-kube-api-access-pj4dw\") pod \"isvc-canary-create-predictor-86d48b7dcf-fvqfw\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") " pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:13.856538 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:13.856509    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:14.038687 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:14.038600    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" event={"ID":"1747489c-7deb-499b-a98d-4b86fa70a15a","Type":"ContainerStarted","Data":"4a459a35def8bde2af450dbdcfa43be02cad0d8b5eff6ccf18904ed1183ef406"}
Oct 30 22:37:14.038687 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:14.038657    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" event={"ID":"1747489c-7deb-499b-a98d-4b86fa70a15a","Type":"ContainerStarted","Data":"66f1b69308d1c1500aa745a05cc8ccc0e7adc05ce132e06f4c0a1d95eb2fade6"}
Oct 30 22:37:14.284797 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:14.284774    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"]
Oct 30 22:37:14.287244 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:37:14.287213    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3304e79c_2498_4fc7_b4f4_093f404783f5.slice/crio-21a0128ad5a3f519930a718e8b750c59d3d233b2ce72c536135501cd3d734a08 WatchSource:0}: Error finding container 21a0128ad5a3f519930a718e8b750c59d3d233b2ce72c536135501cd3d734a08: Status 404 returned error can't find the container with id 21a0128ad5a3f519930a718e8b750c59d3d233b2ce72c536135501cd3d734a08
Oct 30 22:37:15.044004 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:15.043963    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" event={"ID":"3304e79c-2498-4fc7-b4f4-093f404783f5","Type":"ContainerStarted","Data":"0f91b196b4920ce997205867f1b864c71bbdfd991f4c1caaa69f0283241be204"}
Oct 30 22:37:15.044428 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:15.044031    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" event={"ID":"3304e79c-2498-4fc7-b4f4-093f404783f5","Type":"ContainerStarted","Data":"21a0128ad5a3f519930a718e8b750c59d3d233b2ce72c536135501cd3d734a08"}
Oct 30 22:37:18.059485 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:18.059450    4473 generic.go:356] "Generic (PLEG): container finished" podID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerID="0f91b196b4920ce997205867f1b864c71bbdfd991f4c1caaa69f0283241be204" exitCode=0
Oct 30 22:37:18.059870 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:18.059525    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" event={"ID":"3304e79c-2498-4fc7-b4f4-093f404783f5","Type":"ContainerDied","Data":"0f91b196b4920ce997205867f1b864c71bbdfd991f4c1caaa69f0283241be204"}
Oct 30 22:37:18.061228 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:18.061128    4473 generic.go:356] "Generic (PLEG): container finished" podID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerID="4a459a35def8bde2af450dbdcfa43be02cad0d8b5eff6ccf18904ed1183ef406" exitCode=0
Oct 30 22:37:18.061228 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:18.061189    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" event={"ID":"1747489c-7deb-499b-a98d-4b86fa70a15a","Type":"ContainerDied","Data":"4a459a35def8bde2af450dbdcfa43be02cad0d8b5eff6ccf18904ed1183ef406"}
Oct 30 22:37:19.068030 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:19.067984    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" event={"ID":"1747489c-7deb-499b-a98d-4b86fa70a15a","Type":"ContainerStarted","Data":"0e4df46ae379e0795e62b35bdfc32e621dcd1094caea8991ee55a280858b33cf"}
Oct 30 22:37:19.068528 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:19.068074    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" event={"ID":"1747489c-7deb-499b-a98d-4b86fa70a15a","Type":"ContainerStarted","Data":"288d0daf7a8483df609218ec91677638f7ad0f6dc0b1abec3b57d03dbe45bcb0"}
Oct 30 22:37:19.068528 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:19.068459    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:19.068764 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:19.068745    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:19.070284 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:19.070259    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Oct 30 22:37:19.070863 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:19.070842    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" event={"ID":"3304e79c-2498-4fc7-b4f4-093f404783f5","Type":"ContainerStarted","Data":"f892284377862d60164fa9a3b5e987349cee6191220cb64c1483b247ca3b9463"}
Oct 30 22:37:19.071001 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:19.070888    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" event={"ID":"3304e79c-2498-4fc7-b4f4-093f404783f5","Type":"ContainerStarted","Data":"5e759843fa265ed57a776588e411409cc240fe296b927a592eb14945962c85eb"}
Oct 30 22:37:19.071118 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:19.071103    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:19.188128 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:19.188078    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" podStartSLOduration=3.27824141 podStartE2EDuration="7.188064529s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.909823119s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:19.187443361 +0000 UTC m=+2657.423652707" watchObservedRunningTime="2026-10-30 22:37:19.188064529 +0000 UTC m=+2657.424273874"
Oct 30 22:37:19.307675 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:19.307627    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" podStartSLOduration=2.657568612 podStartE2EDuration="6.307613195s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.650044583s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:19.306251832 +0000 UTC m=+2657.542461178" watchObservedRunningTime="2026-10-30 22:37:19.307613195 +0000 UTC m=+2657.543822540"
Oct 30 22:37:20.075510 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:20.075469    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Oct 30 22:37:20.076000 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:20.075773    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:20.076900 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:20.076876    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Oct 30 22:37:21.080031 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:21.079990    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Oct 30 22:37:21.080429 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:21.080073    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Oct 30 22:37:26.084763 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:26.084733    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:37:26.085244 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:26.085182    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:37:26.085244 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:26.085205    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Oct 30 22:37:26.085617 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:26.085593    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Oct 30 22:37:35.987457 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:35.987427    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:37:36.085900 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:36.085864    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Oct 30 22:37:36.086066 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:36.085864    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Oct 30 22:37:41.303627 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:37:41.303593    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:37:42.535777 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:42.535747    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"]
Oct 30 22:37:42.536248 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:42.535990    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h" podUID="48a8c8f1-3b5a-4515-8327-9acea448c6d3" containerName="kserve-container" containerID="cri-o://13111bd285051b16372d97e7d1f36fdbcb905f4f1f2cfa555ccdfa519d5f7e24" gracePeriod=30
Oct 30 22:37:42.536248 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:42.536058    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h" podUID="48a8c8f1-3b5a-4515-8327-9acea448c6d3" containerName="kube-rbac-proxy" containerID="cri-o://0ccbc5d04161b4c30c5a3a6261df3bcadc22a1ea27b4edcefa6c55f033ac1af4" gracePeriod=30
Oct 30 22:37:42.570454 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:42.570418    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:43.177514 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:43.177479    4473 generic.go:356] "Generic (PLEG): container finished" podID="48a8c8f1-3b5a-4515-8327-9acea448c6d3" containerID="0ccbc5d04161b4c30c5a3a6261df3bcadc22a1ea27b4edcefa6c55f033ac1af4" exitCode=0
Oct 30 22:37:43.177703 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:43.177550    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h" event={"ID":"48a8c8f1-3b5a-4515-8327-9acea448c6d3","Type":"ContainerDied","Data":"0ccbc5d04161b4c30c5a3a6261df3bcadc22a1ea27b4edcefa6c55f033ac1af4"}
Oct 30 22:37:45.978541 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:45.978498    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h" podUID="48a8c8f1-3b5a-4515-8327-9acea448c6d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Oct 30 22:37:45.984975 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:45.984929    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h" podUID="48a8c8f1-3b5a-4515-8327-9acea448c6d3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.53:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.134.0.53:8080: connect: connection refused"
Oct 30 22:37:46.085606 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:46.085566    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Oct 30 22:37:46.085768 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:46.085572    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Oct 30 22:37:49.204352 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.204287    4473 generic.go:356] "Generic (PLEG): container finished" podID="48a8c8f1-3b5a-4515-8327-9acea448c6d3" containerID="13111bd285051b16372d97e7d1f36fdbcb905f4f1f2cfa555ccdfa519d5f7e24" exitCode=0
Oct 30 22:37:49.204684 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.204362    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h" event={"ID":"48a8c8f1-3b5a-4515-8327-9acea448c6d3","Type":"ContainerDied","Data":"13111bd285051b16372d97e7d1f36fdbcb905f4f1f2cfa555ccdfa519d5f7e24"}
Oct 30 22:37:49.576325 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.576291    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:37:49.655412 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.655377    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48a8c8f1-3b5a-4515-8327-9acea448c6d3-proxy-tls\" (UniqueName: \"kubernetes.io/secret/48a8c8f1-3b5a-4515-8327-9acea448c6d3-proxy-tls\") pod \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") "
Oct 30 22:37:49.655603 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.655440    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kserve-provision-location\") pod \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") "
Oct 30 22:37:49.655603 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.655518    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kube-api-access-lp9g7\" (UniqueName: \"kubernetes.io/projected/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kube-api-access-lp9g7\") pod \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") "
Oct 30 22:37:49.655603 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.655549    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a8c8f1-3b5a-4515-8327-9acea448c6d3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48a8c8f1-3b5a-4515-8327-9acea448c6d3-cabundle-cert\") pod \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") "
Oct 30 22:37:49.655603 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.655582    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kube-rbac-proxy-sar-config\") pod \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\" (UID: \"48a8c8f1-3b5a-4515-8327-9acea448c6d3\") "
Oct 30 22:37:49.655905 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.655876    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kserve-provision-location" pod "48a8c8f1-3b5a-4515-8327-9acea448c6d3" (UID: "48a8c8f1-3b5a-4515-8327-9acea448c6d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:49.656020 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.655997    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kube-rbac-proxy-sar-config" pod "48a8c8f1-3b5a-4515-8327-9acea448c6d3" (UID: "48a8c8f1-3b5a-4515-8327-9acea448c6d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:49.656091 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.655993    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a8c8f1-3b5a-4515-8327-9acea448c6d3-cabundle-cert" pod "48a8c8f1-3b5a-4515-8327-9acea448c6d3" (UID: "48a8c8f1-3b5a-4515-8327-9acea448c6d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:49.657706 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.657686    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kube-api-access-lp9g7" pod "48a8c8f1-3b5a-4515-8327-9acea448c6d3" (UID: "48a8c8f1-3b5a-4515-8327-9acea448c6d3"). InnerVolumeSpecName "kube-api-access-lp9g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:49.657813 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.657735    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a8c8f1-3b5a-4515-8327-9acea448c6d3-proxy-tls" pod "48a8c8f1-3b5a-4515-8327-9acea448c6d3" (UID: "48a8c8f1-3b5a-4515-8327-9acea448c6d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:37:49.756701 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.756624    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48a8c8f1-3b5a-4515-8327-9acea448c6d3-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:49.756701 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.756651    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:49.756701 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.756662    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp9g7\" (UniqueName: \"kubernetes.io/projected/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kube-api-access-lp9g7\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:49.756701 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.756671    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48a8c8f1-3b5a-4515-8327-9acea448c6d3-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:49.756701 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:49.756680    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48a8c8f1-3b5a-4515-8327-9acea448c6d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:50.216630 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:50.216586    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h" event={"ID":"48a8c8f1-3b5a-4515-8327-9acea448c6d3","Type":"ContainerDied","Data":"931133140dc6dc1a865c306709a10200fbf719527aad35a2570b0fbd8ced87ae"}
Oct 30 22:37:50.217146 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:50.216681    4473 scope.go:122] "RemoveContainer" containerID="0ccbc5d04161b4c30c5a3a6261df3bcadc22a1ea27b4edcefa6c55f033ac1af4"
Oct 30 22:37:50.217146 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:50.216599    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"
Oct 30 22:37:50.224962 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:50.224935    4473 scope.go:122] "RemoveContainer" containerID="13111bd285051b16372d97e7d1f36fdbcb905f4f1f2cfa555ccdfa519d5f7e24"
Oct 30 22:37:50.232517 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:50.232500    4473 scope.go:122] "RemoveContainer" containerID="203e34c62ae74c8c0e6ab8ead98b0a87963e4818081bafe2710b01e45b9b823a"
Oct 30 22:37:50.435163 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:50.435119    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"]
Oct 30 22:37:50.485698 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:50.485625    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/xgboost-v2-mlserver-predictor-78bcc674db-gkk8h"]
Oct 30 22:37:51.887193 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:51.887165    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a8c8f1-3b5a-4515-8327-9acea448c6d3" path="/var/lib/kubelet/pods/48a8c8f1-3b5a-4515-8327-9acea448c6d3/volumes"
Oct 30 22:37:53.113092 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.113052    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:53.319633 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.319585    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:53.598489 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.598453    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"]
Oct 30 22:37:53.598884 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.598866    4473 cpu_manager.go:414] "containerMap: removing container" podUID="48a8c8f1-3b5a-4515-8327-9acea448c6d3" containerName="kube-rbac-proxy"
Oct 30 22:37:53.598997 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.598885    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a8c8f1-3b5a-4515-8327-9acea448c6d3" containerName="kube-rbac-proxy"
Oct 30 22:37:53.598997 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.598906    4473 cpu_manager.go:414] "containerMap: removing container" podUID="48a8c8f1-3b5a-4515-8327-9acea448c6d3" containerName="storage-initializer"
Oct 30 22:37:53.598997 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.598913    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a8c8f1-3b5a-4515-8327-9acea448c6d3" containerName="storage-initializer"
Oct 30 22:37:53.598997 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.598923    4473 cpu_manager.go:414] "containerMap: removing container" podUID="48a8c8f1-3b5a-4515-8327-9acea448c6d3" containerName="kserve-container"
Oct 30 22:37:53.598997 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.598929    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a8c8f1-3b5a-4515-8327-9acea448c6d3" containerName="kserve-container"
Oct 30 22:37:53.599256 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.599034    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a8c8f1-3b5a-4515-8327-9acea448c6d3" containerName="kube-rbac-proxy"
Oct 30 22:37:53.599256 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.599050    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a8c8f1-3b5a-4515-8327-9acea448c6d3" containerName="kserve-container"
Oct 30 22:37:53.604056 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.604035    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:37:53.657060 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.657028    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Oct 30 22:37:53.688421 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.688391    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-5789f96589-lb5dl\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") " pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:37:53.688421 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.688422    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-5789f96589-lb5dl\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") " pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:37:53.688606 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.688447    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x94w\" (UniqueName: \"kubernetes.io/projected/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kube-api-access-8x94w\") pod \"isvc-xgboost-runtime-predictor-5789f96589-lb5dl\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") " pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:37:53.688606 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.688559    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-5789f96589-lb5dl\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") " pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:37:53.688671 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.688590    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-5789f96589-lb5dl\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") " pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:37:53.707370 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.707344    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:37:53.707549 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.707351    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:37:53.707549 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.707401    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:37:53.707758 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.707425    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:37:53.707758 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.707432    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Oct 30 22:37:53.707758 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.707456    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:37:53.709194 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.709171    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"]
Oct 30 22:37:53.789356 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.789319    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-5789f96589-lb5dl\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") " pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:37:53.789356 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.789353    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-5789f96589-lb5dl\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") " pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:37:53.789631 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.789378    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x94w\" (UniqueName: \"kubernetes.io/projected/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kube-api-access-8x94w\") pod \"isvc-xgboost-runtime-predictor-5789f96589-lb5dl\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") " pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:37:53.789631 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.789413    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-5789f96589-lb5dl\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") " pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:37:53.789631 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.789431    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-5789f96589-lb5dl\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") " pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:37:53.789816 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.789788    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-5789f96589-lb5dl\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") " pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:37:53.790078 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.790059    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-5789f96589-lb5dl\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") " pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:37:53.790131 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.790096    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-5789f96589-lb5dl\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") " pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:37:53.791976 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.791941    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-5789f96589-lb5dl\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") " pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:37:53.847349 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.847322    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x94w\" (UniqueName: \"kubernetes.io/projected/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kube-api-access-8x94w\") pod \"isvc-xgboost-runtime-predictor-5789f96589-lb5dl\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") " pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:37:53.921428 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:53.921399    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:37:54.370998 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:54.370975    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"]
Oct 30 22:37:54.373682 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:37:54.373652    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0616b1c_0d0b_449d_9ee7_e4bd87dbcef9.slice/crio-2bb5efac216046b4325b993132712dbb8498135322661f2f6764f6593dade05c WatchSource:0}: Error finding container 2bb5efac216046b4325b993132712dbb8498135322661f2f6764f6593dade05c: Status 404 returned error can't find the container with id 2bb5efac216046b4325b993132712dbb8498135322661f2f6764f6593dade05c
Oct 30 22:37:55.240736 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:55.240698    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" event={"ID":"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9","Type":"ContainerStarted","Data":"9ae3dee78211daf0bf882f7a180e7f7721def6e1e06d26d0b2528ad21ca1f18d"}
Oct 30 22:37:55.240917 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:55.240786    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" event={"ID":"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9","Type":"ContainerStarted","Data":"2bb5efac216046b4325b993132712dbb8498135322661f2f6764f6593dade05c"}
Oct 30 22:37:56.085378 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:56.085342    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Oct 30 22:37:56.085760 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:56.085580    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Oct 30 22:37:59.259191 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:59.259158    4473 generic.go:356] "Generic (PLEG): container finished" podID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerID="9ae3dee78211daf0bf882f7a180e7f7721def6e1e06d26d0b2528ad21ca1f18d" exitCode=0
Oct 30 22:37:59.259618 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:59.259217    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" event={"ID":"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9","Type":"ContainerDied","Data":"9ae3dee78211daf0bf882f7a180e7f7721def6e1e06d26d0b2528ad21ca1f18d"}
Oct 30 22:37:59.260502 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:37:59.260486    4473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:38:02.130659 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:02.130618    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:38:02.135439 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:02.135421    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:38:06.085223 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:06.085181    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Oct 30 22:38:06.085687 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:06.085635    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Oct 30 22:38:06.511341 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:06.511299    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:11.306484 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:38:11.306445    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:38:16.085252 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:16.085209    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Oct 30 22:38:16.085803 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:16.085675    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Oct 30 22:38:19.355018 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:19.354983    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" event={"ID":"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9","Type":"ContainerStarted","Data":"9e4cbdc5eb51fec8ebc30d32f581719b8dae14035e4a512422a629e0caa737c4"}
Oct 30 22:38:19.355446 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:19.355046    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" event={"ID":"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9","Type":"ContainerStarted","Data":"fff2bd4a3c1af7694fab03cbe30e94c9c1b5f05ba33fa51d9fcecd6d30f98f06"}
Oct 30 22:38:19.355446 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:19.355222    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:38:19.526347 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:19.526295    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" podStartSLOduration=3.392290591 podStartE2EDuration="26.526281299s" totalImagesPullingTime="19.184261043s" totalInitContainerRuntime="3.949729665s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:19.525253883 +0000 UTC m=+2717.761463229" watchObservedRunningTime="2026-10-30 22:38:19.526281299 +0000 UTC m=+2717.762490643"
Oct 30 22:38:20.365648 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:20.365611    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:38:20.367406 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:20.367369    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Oct 30 22:38:21.371266 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:21.371222    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Oct 30 22:38:26.085878 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:26.085843    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Oct 30 22:38:26.086300 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:26.085845    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Oct 30 22:38:26.376800 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:26.376720    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:38:26.377151 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:26.377123    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Oct 30 22:38:36.085203 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:36.085158    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Oct 30 22:38:36.085685 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:36.085633    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Oct 30 22:38:36.377411 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:36.377332    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Oct 30 22:38:40.881384 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:40.881331    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Oct 30 22:38:40.881787 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:40.881344    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Oct 30 22:38:41.303565 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:38:41.303529    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:38:46.378008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:46.377902    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Oct 30 22:38:50.882117 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:50.882079    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:38:50.882664 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:50.882187    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:38:56.377185 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:38:56.377147    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Oct 30 22:39:02.984712 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:02.984670    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:02.999481 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:02.999455    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"]
Oct 30 22:39:02.999705 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:02.999680    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kserve-container" containerID="cri-o://288d0daf7a8483df609218ec91677638f7ad0f6dc0b1abec3b57d03dbe45bcb0" gracePeriod=30
Oct 30 22:39:02.999799 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:02.999732    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kube-rbac-proxy" containerID="cri-o://0e4df46ae379e0795e62b35bdfc32e621dcd1094caea8991ee55a280858b33cf" gracePeriod=30
Oct 30 22:39:03.042159 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:03.042130    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"]
Oct 30 22:39:03.042403 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:03.042363    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kserve-container" containerID="cri-o://5e759843fa265ed57a776588e411409cc240fe296b927a592eb14945962c85eb" gracePeriod=30
Oct 30 22:39:03.042506 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:03.042380    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kube-rbac-proxy" containerID="cri-o://f892284377862d60164fa9a3b5e987349cee6191220cb64c1483b247ca3b9463" gracePeriod=30
Oct 30 22:39:03.145062 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:03.145033    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:03.547014 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:03.546982    4473 generic.go:356] "Generic (PLEG): container finished" podID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerID="f892284377862d60164fa9a3b5e987349cee6191220cb64c1483b247ca3b9463" exitCode=0
Oct 30 22:39:03.547208 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:03.547057    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" event={"ID":"3304e79c-2498-4fc7-b4f4-093f404783f5","Type":"ContainerDied","Data":"f892284377862d60164fa9a3b5e987349cee6191220cb64c1483b247ca3b9463"}
Oct 30 22:39:03.549197 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:03.549176    4473 generic.go:356] "Generic (PLEG): container finished" podID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerID="0e4df46ae379e0795e62b35bdfc32e621dcd1094caea8991ee55a280858b33cf" exitCode=0
Oct 30 22:39:03.549314 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:03.549224    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" event={"ID":"1747489c-7deb-499b-a98d-4b86fa70a15a","Type":"ContainerDied","Data":"0e4df46ae379e0795e62b35bdfc32e621dcd1094caea8991ee55a280858b33cf"}
Oct 30 22:39:06.080840 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:06.080793    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Oct 30 22:39:06.081272 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:06.080797    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Oct 30 22:39:06.377628 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:06.377544    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Oct 30 22:39:06.873543 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:06.873510    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:08.206344 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.206308    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:39:08.214017 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.213998    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:39:08.319281 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.319208    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1747489c-7deb-499b-a98d-4b86fa70a15a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1747489c-7deb-499b-a98d-4b86fa70a15a-kube-rbac-proxy-sar-config\") pod \"1747489c-7deb-499b-a98d-4b86fa70a15a\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") "
Oct 30 22:39:08.319281 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.319241    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3304e79c-2498-4fc7-b4f4-093f404783f5-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3304e79c-2498-4fc7-b4f4-093f404783f5-kserve-provision-location\") pod \"3304e79c-2498-4fc7-b4f4-093f404783f5\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") "
Oct 30 22:39:08.319281 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.319265    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3304e79c-2498-4fc7-b4f4-093f404783f5-proxy-tls\" (UniqueName: \"kubernetes.io/secret/3304e79c-2498-4fc7-b4f4-093f404783f5-proxy-tls\") pod \"3304e79c-2498-4fc7-b4f4-093f404783f5\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") "
Oct 30 22:39:08.319522 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.319296    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1747489c-7deb-499b-a98d-4b86fa70a15a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1747489c-7deb-499b-a98d-4b86fa70a15a-cabundle-cert\") pod \"1747489c-7deb-499b-a98d-4b86fa70a15a\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") "
Oct 30 22:39:08.319522 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.319327    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1747489c-7deb-499b-a98d-4b86fa70a15a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1747489c-7deb-499b-a98d-4b86fa70a15a-kserve-provision-location\") pod \"1747489c-7deb-499b-a98d-4b86fa70a15a\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") "
Oct 30 22:39:08.319522 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.319357    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1747489c-7deb-499b-a98d-4b86fa70a15a-kube-api-access-bf98x\" (UniqueName: \"kubernetes.io/projected/1747489c-7deb-499b-a98d-4b86fa70a15a-kube-api-access-bf98x\") pod \"1747489c-7deb-499b-a98d-4b86fa70a15a\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") "
Oct 30 22:39:08.319522 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.319417    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3304e79c-2498-4fc7-b4f4-093f404783f5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3304e79c-2498-4fc7-b4f4-093f404783f5-kube-rbac-proxy-sar-config\") pod \"3304e79c-2498-4fc7-b4f4-093f404783f5\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") "
Oct 30 22:39:08.319709 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.319566    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3304e79c-2498-4fc7-b4f4-093f404783f5-kserve-provision-location" pod "3304e79c-2498-4fc7-b4f4-093f404783f5" (UID: "3304e79c-2498-4fc7-b4f4-093f404783f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:08.319709 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.319693    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1747489c-7deb-499b-a98d-4b86fa70a15a-kserve-provision-location" pod "1747489c-7deb-499b-a98d-4b86fa70a15a" (UID: "1747489c-7deb-499b-a98d-4b86fa70a15a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:08.319791 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.319718    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1747489c-7deb-499b-a98d-4b86fa70a15a-cabundle-cert" pod "1747489c-7deb-499b-a98d-4b86fa70a15a" (UID: "1747489c-7deb-499b-a98d-4b86fa70a15a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:08.319828 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.319807    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3304e79c-2498-4fc7-b4f4-093f404783f5-kube-rbac-proxy-sar-config" pod "3304e79c-2498-4fc7-b4f4-093f404783f5" (UID: "3304e79c-2498-4fc7-b4f4-093f404783f5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:08.319869 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.319851    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3304e79c-2498-4fc7-b4f4-093f404783f5-kube-api-access-pj4dw\" (UniqueName: \"kubernetes.io/projected/3304e79c-2498-4fc7-b4f4-093f404783f5-kube-api-access-pj4dw\") pod \"3304e79c-2498-4fc7-b4f4-093f404783f5\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") "
Oct 30 22:39:08.319905 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.319887    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1747489c-7deb-499b-a98d-4b86fa70a15a-kube-rbac-proxy-sar-config" pod "1747489c-7deb-499b-a98d-4b86fa70a15a" (UID: "1747489c-7deb-499b-a98d-4b86fa70a15a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:08.320166 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.320017    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1747489c-7deb-499b-a98d-4b86fa70a15a-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1747489c-7deb-499b-a98d-4b86fa70a15a-proxy-tls\") pod \"1747489c-7deb-499b-a98d-4b86fa70a15a\" (UID: \"1747489c-7deb-499b-a98d-4b86fa70a15a\") "
Oct 30 22:39:08.320166 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.320131    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3304e79c-2498-4fc7-b4f4-093f404783f5-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3304e79c-2498-4fc7-b4f4-093f404783f5-cabundle-cert\") pod \"3304e79c-2498-4fc7-b4f4-093f404783f5\" (UID: \"3304e79c-2498-4fc7-b4f4-093f404783f5\") "
Oct 30 22:39:08.320581 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.320525    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3304e79c-2498-4fc7-b4f4-093f404783f5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:08.320581 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.320545    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1747489c-7deb-499b-a98d-4b86fa70a15a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:08.320581 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.320560    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3304e79c-2498-4fc7-b4f4-093f404783f5-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:08.320581 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.320574    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1747489c-7deb-499b-a98d-4b86fa70a15a-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:08.320803 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.320589    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1747489c-7deb-499b-a98d-4b86fa70a15a-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:08.320803 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.320674    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3304e79c-2498-4fc7-b4f4-093f404783f5-cabundle-cert" pod "3304e79c-2498-4fc7-b4f4-093f404783f5" (UID: "3304e79c-2498-4fc7-b4f4-093f404783f5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:08.321611 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.321584    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3304e79c-2498-4fc7-b4f4-093f404783f5-proxy-tls" pod "3304e79c-2498-4fc7-b4f4-093f404783f5" (UID: "3304e79c-2498-4fc7-b4f4-093f404783f5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:08.321862 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.321838    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3304e79c-2498-4fc7-b4f4-093f404783f5-kube-api-access-pj4dw" pod "3304e79c-2498-4fc7-b4f4-093f404783f5" (UID: "3304e79c-2498-4fc7-b4f4-093f404783f5"). InnerVolumeSpecName "kube-api-access-pj4dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:08.321862 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.321847    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1747489c-7deb-499b-a98d-4b86fa70a15a-kube-api-access-bf98x" pod "1747489c-7deb-499b-a98d-4b86fa70a15a" (UID: "1747489c-7deb-499b-a98d-4b86fa70a15a"). InnerVolumeSpecName "kube-api-access-bf98x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:08.322019 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.321910    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1747489c-7deb-499b-a98d-4b86fa70a15a-proxy-tls" pod "1747489c-7deb-499b-a98d-4b86fa70a15a" (UID: "1747489c-7deb-499b-a98d-4b86fa70a15a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:08.421496 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.421467    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj4dw\" (UniqueName: \"kubernetes.io/projected/3304e79c-2498-4fc7-b4f4-093f404783f5-kube-api-access-pj4dw\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:08.421496 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.421490    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1747489c-7deb-499b-a98d-4b86fa70a15a-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:08.421496 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.421502    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3304e79c-2498-4fc7-b4f4-093f404783f5-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:08.421723 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.421511    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3304e79c-2498-4fc7-b4f4-093f404783f5-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:08.421723 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.421520    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf98x\" (UniqueName: \"kubernetes.io/projected/1747489c-7deb-499b-a98d-4b86fa70a15a-kube-api-access-bf98x\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:08.573389 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.573302    4473 generic.go:356] "Generic (PLEG): container finished" podID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerID="5e759843fa265ed57a776588e411409cc240fe296b927a592eb14945962c85eb" exitCode=0
Oct 30 22:39:08.573389 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.573362    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"
Oct 30 22:39:08.573622 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.573389    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" event={"ID":"3304e79c-2498-4fc7-b4f4-093f404783f5","Type":"ContainerDied","Data":"5e759843fa265ed57a776588e411409cc240fe296b927a592eb14945962c85eb"}
Oct 30 22:39:08.573622 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.573490    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw" event={"ID":"3304e79c-2498-4fc7-b4f4-093f404783f5","Type":"ContainerDied","Data":"21a0128ad5a3f519930a718e8b750c59d3d233b2ce72c536135501cd3d734a08"}
Oct 30 22:39:08.573622 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.573528    4473 scope.go:122] "RemoveContainer" containerID="f892284377862d60164fa9a3b5e987349cee6191220cb64c1483b247ca3b9463"
Oct 30 22:39:08.575473 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.575448    4473 generic.go:356] "Generic (PLEG): container finished" podID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerID="288d0daf7a8483df609218ec91677638f7ad0f6dc0b1abec3b57d03dbe45bcb0" exitCode=0
Oct 30 22:39:08.575580 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.575502    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"
Oct 30 22:39:08.575580 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.575534    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" event={"ID":"1747489c-7deb-499b-a98d-4b86fa70a15a","Type":"ContainerDied","Data":"288d0daf7a8483df609218ec91677638f7ad0f6dc0b1abec3b57d03dbe45bcb0"}
Oct 30 22:39:08.575670 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.575589    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m" event={"ID":"1747489c-7deb-499b-a98d-4b86fa70a15a","Type":"ContainerDied","Data":"66f1b69308d1c1500aa745a05cc8ccc0e7adc05ce132e06f4c0a1d95eb2fade6"}
Oct 30 22:39:08.583397 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.583381    4473 scope.go:122] "RemoveContainer" containerID="5e759843fa265ed57a776588e411409cc240fe296b927a592eb14945962c85eb"
Oct 30 22:39:08.591182 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.591165    4473 scope.go:122] "RemoveContainer" containerID="0f91b196b4920ce997205867f1b864c71bbdfd991f4c1caaa69f0283241be204"
Oct 30 22:39:08.598467 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.598452    4473 scope.go:122] "RemoveContainer" containerID="f892284377862d60164fa9a3b5e987349cee6191220cb64c1483b247ca3b9463"
Oct 30 22:39:08.598726 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:39:08.598705    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f892284377862d60164fa9a3b5e987349cee6191220cb64c1483b247ca3b9463\": container with ID starting with f892284377862d60164fa9a3b5e987349cee6191220cb64c1483b247ca3b9463 not found: ID does not exist" containerID="f892284377862d60164fa9a3b5e987349cee6191220cb64c1483b247ca3b9463"
Oct 30 22:39:08.598821 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.598733    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f892284377862d60164fa9a3b5e987349cee6191220cb64c1483b247ca3b9463"} err="failed to get container status \"f892284377862d60164fa9a3b5e987349cee6191220cb64c1483b247ca3b9463\": rpc error: code = NotFound desc = could not find container \"f892284377862d60164fa9a3b5e987349cee6191220cb64c1483b247ca3b9463\": container with ID starting with f892284377862d60164fa9a3b5e987349cee6191220cb64c1483b247ca3b9463 not found: ID does not exist"
Oct 30 22:39:08.598821 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.598749    4473 scope.go:122] "RemoveContainer" containerID="5e759843fa265ed57a776588e411409cc240fe296b927a592eb14945962c85eb"
Oct 30 22:39:08.598979 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:39:08.598943    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e759843fa265ed57a776588e411409cc240fe296b927a592eb14945962c85eb\": container with ID starting with 5e759843fa265ed57a776588e411409cc240fe296b927a592eb14945962c85eb not found: ID does not exist" containerID="5e759843fa265ed57a776588e411409cc240fe296b927a592eb14945962c85eb"
Oct 30 22:39:08.599028 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.598988    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e759843fa265ed57a776588e411409cc240fe296b927a592eb14945962c85eb"} err="failed to get container status \"5e759843fa265ed57a776588e411409cc240fe296b927a592eb14945962c85eb\": rpc error: code = NotFound desc = could not find container \"5e759843fa265ed57a776588e411409cc240fe296b927a592eb14945962c85eb\": container with ID starting with 5e759843fa265ed57a776588e411409cc240fe296b927a592eb14945962c85eb not found: ID does not exist"
Oct 30 22:39:08.599028 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.599003    4473 scope.go:122] "RemoveContainer" containerID="0f91b196b4920ce997205867f1b864c71bbdfd991f4c1caaa69f0283241be204"
Oct 30 22:39:08.599207 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:39:08.599191    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f91b196b4920ce997205867f1b864c71bbdfd991f4c1caaa69f0283241be204\": container with ID starting with 0f91b196b4920ce997205867f1b864c71bbdfd991f4c1caaa69f0283241be204 not found: ID does not exist" containerID="0f91b196b4920ce997205867f1b864c71bbdfd991f4c1caaa69f0283241be204"
Oct 30 22:39:08.599248 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.599212    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f91b196b4920ce997205867f1b864c71bbdfd991f4c1caaa69f0283241be204"} err="failed to get container status \"0f91b196b4920ce997205867f1b864c71bbdfd991f4c1caaa69f0283241be204\": rpc error: code = NotFound desc = could not find container \"0f91b196b4920ce997205867f1b864c71bbdfd991f4c1caaa69f0283241be204\": container with ID starting with 0f91b196b4920ce997205867f1b864c71bbdfd991f4c1caaa69f0283241be204 not found: ID does not exist"
Oct 30 22:39:08.599248 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.599225    4473 scope.go:122] "RemoveContainer" containerID="0e4df46ae379e0795e62b35bdfc32e621dcd1094caea8991ee55a280858b33cf"
Oct 30 22:39:08.606269 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.606253    4473 scope.go:122] "RemoveContainer" containerID="288d0daf7a8483df609218ec91677638f7ad0f6dc0b1abec3b57d03dbe45bcb0"
Oct 30 22:39:08.613136 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.613122    4473 scope.go:122] "RemoveContainer" containerID="4a459a35def8bde2af450dbdcfa43be02cad0d8b5eff6ccf18904ed1183ef406"
Oct 30 22:39:08.620079 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.620064    4473 scope.go:122] "RemoveContainer" containerID="0e4df46ae379e0795e62b35bdfc32e621dcd1094caea8991ee55a280858b33cf"
Oct 30 22:39:08.620374 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:39:08.620350    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e4df46ae379e0795e62b35bdfc32e621dcd1094caea8991ee55a280858b33cf\": container with ID starting with 0e4df46ae379e0795e62b35bdfc32e621dcd1094caea8991ee55a280858b33cf not found: ID does not exist" containerID="0e4df46ae379e0795e62b35bdfc32e621dcd1094caea8991ee55a280858b33cf"
Oct 30 22:39:08.620431 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.620382    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e4df46ae379e0795e62b35bdfc32e621dcd1094caea8991ee55a280858b33cf"} err="failed to get container status \"0e4df46ae379e0795e62b35bdfc32e621dcd1094caea8991ee55a280858b33cf\": rpc error: code = NotFound desc = could not find container \"0e4df46ae379e0795e62b35bdfc32e621dcd1094caea8991ee55a280858b33cf\": container with ID starting with 0e4df46ae379e0795e62b35bdfc32e621dcd1094caea8991ee55a280858b33cf not found: ID does not exist"
Oct 30 22:39:08.620431 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.620398    4473 scope.go:122] "RemoveContainer" containerID="288d0daf7a8483df609218ec91677638f7ad0f6dc0b1abec3b57d03dbe45bcb0"
Oct 30 22:39:08.620655 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:39:08.620637    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288d0daf7a8483df609218ec91677638f7ad0f6dc0b1abec3b57d03dbe45bcb0\": container with ID starting with 288d0daf7a8483df609218ec91677638f7ad0f6dc0b1abec3b57d03dbe45bcb0 not found: ID does not exist" containerID="288d0daf7a8483df609218ec91677638f7ad0f6dc0b1abec3b57d03dbe45bcb0"
Oct 30 22:39:08.620705 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.620660    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288d0daf7a8483df609218ec91677638f7ad0f6dc0b1abec3b57d03dbe45bcb0"} err="failed to get container status \"288d0daf7a8483df609218ec91677638f7ad0f6dc0b1abec3b57d03dbe45bcb0\": rpc error: code = NotFound desc = could not find container \"288d0daf7a8483df609218ec91677638f7ad0f6dc0b1abec3b57d03dbe45bcb0\": container with ID starting with 288d0daf7a8483df609218ec91677638f7ad0f6dc0b1abec3b57d03dbe45bcb0 not found: ID does not exist"
Oct 30 22:39:08.620705 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.620675    4473 scope.go:122] "RemoveContainer" containerID="4a459a35def8bde2af450dbdcfa43be02cad0d8b5eff6ccf18904ed1183ef406"
Oct 30 22:39:08.620902 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:39:08.620883    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a459a35def8bde2af450dbdcfa43be02cad0d8b5eff6ccf18904ed1183ef406\": container with ID starting with 4a459a35def8bde2af450dbdcfa43be02cad0d8b5eff6ccf18904ed1183ef406 not found: ID does not exist" containerID="4a459a35def8bde2af450dbdcfa43be02cad0d8b5eff6ccf18904ed1183ef406"
Oct 30 22:39:08.621004 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.620917    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a459a35def8bde2af450dbdcfa43be02cad0d8b5eff6ccf18904ed1183ef406"} err="failed to get container status \"4a459a35def8bde2af450dbdcfa43be02cad0d8b5eff6ccf18904ed1183ef406\": rpc error: code = NotFound desc = could not find container \"4a459a35def8bde2af450dbdcfa43be02cad0d8b5eff6ccf18904ed1183ef406\": container with ID starting with 4a459a35def8bde2af450dbdcfa43be02cad0d8b5eff6ccf18904ed1183ef406 not found: ID does not exist"
Oct 30 22:39:08.742506 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.742474    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"]
Oct 30 22:39:08.745996 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.745939    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-create-predictor-86d48b7dcf-fvqfw"]
Oct 30 22:39:08.857799 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.857712    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"]
Oct 30 22:39:08.861613 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:08.861589    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-create-v2-predictor-5ddb8cc8c4-5k58m"]
Oct 30 22:39:09.886345 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:09.886316    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" path="/var/lib/kubelet/pods/1747489c-7deb-499b-a98d-4b86fa70a15a/volumes"
Oct 30 22:39:09.886805 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:09.886791    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" path="/var/lib/kubelet/pods/3304e79c-2498-4fc7-b4f4-093f404783f5/volumes"
Oct 30 22:39:11.303995 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:39:11.303934    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:39:11.665361 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:11.665329    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:11.885286 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:11.885252    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:12.157804 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.157769    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"]
Oct 30 22:39:12.158374 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.158353    4473 cpu_manager.go:414] "containerMap: removing container" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kube-rbac-proxy"
Oct 30 22:39:12.158374 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.158374    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kube-rbac-proxy"
Oct 30 22:39:12.158521 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.158391    4473 cpu_manager.go:414] "containerMap: removing container" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="storage-initializer"
Oct 30 22:39:12.158521 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.158399    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="storage-initializer"
Oct 30 22:39:12.158521 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.158412    4473 cpu_manager.go:414] "containerMap: removing container" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kserve-container"
Oct 30 22:39:12.158521 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.158419    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kserve-container"
Oct 30 22:39:12.158521 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.158435    4473 cpu_manager.go:414] "containerMap: removing container" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kube-rbac-proxy"
Oct 30 22:39:12.158521 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.158441    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kube-rbac-proxy"
Oct 30 22:39:12.158521 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.158470    4473 cpu_manager.go:414] "containerMap: removing container" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="storage-initializer"
Oct 30 22:39:12.158521 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.158477    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="storage-initializer"
Oct 30 22:39:12.158521 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.158489    4473 cpu_manager.go:414] "containerMap: removing container" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kserve-container"
Oct 30 22:39:12.158521 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.158496    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kserve-container"
Oct 30 22:39:12.158835 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.158570    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kserve-container"
Oct 30 22:39:12.158835 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.158583    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kube-rbac-proxy"
Oct 30 22:39:12.158835 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.158592    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1747489c-7deb-499b-a98d-4b86fa70a15a" containerName="kserve-container"
Oct 30 22:39:12.158835 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.158600    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3304e79c-2498-4fc7-b4f4-093f404783f5" containerName="kube-rbac-proxy"
Oct 30 22:39:12.161715 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.161698    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:12.206298 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.206271    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:12.215889 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.215867    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Oct 30 22:39:12.216053 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.216020    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:39:12.216302 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.216290    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"kube-root-ca.crt\""
Oct 30 22:39:12.256701 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.256667    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6557\" (UniqueName: \"kubernetes.io/projected/5b0ba578-d584-4bed-92c3-e4a4254a914c-kube-api-access-z6557\") pod \"isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:12.256840 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.256705    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b0ba578-d584-4bed-92c3-e4a4254a914c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:12.256840 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.256813    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b0ba578-d584-4bed-92c3-e4a4254a914c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:12.256911 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.256849    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b0ba578-d584-4bed-92c3-e4a4254a914c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:12.256911 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.256874    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b0ba578-d584-4bed-92c3-e4a4254a914c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:12.265517 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.265453    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"default-dockercfg-4v4s8\""
Oct 30 22:39:12.265517 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.265452    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"openshift-service-ca.crt\""
Oct 30 22:39:12.265517 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.265458    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Oct 30 22:39:12.265517 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.265490    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"seaweedfs-s3-creds\""
Oct 30 22:39:12.267900 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.267879    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"]
Oct 30 22:39:12.357973 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.357922    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b0ba578-d584-4bed-92c3-e4a4254a914c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:12.357973 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.357976    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b0ba578-d584-4bed-92c3-e4a4254a914c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:12.358528 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.358146    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b0ba578-d584-4bed-92c3-e4a4254a914c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:12.358528 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.358225    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6557\" (UniqueName: \"kubernetes.io/projected/5b0ba578-d584-4bed-92c3-e4a4254a914c-kube-api-access-z6557\") pod \"isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:12.358528 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.358251    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b0ba578-d584-4bed-92c3-e4a4254a914c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:12.358700 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.358629    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b0ba578-d584-4bed-92c3-e4a4254a914c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:12.358822 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.358798    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b0ba578-d584-4bed-92c3-e4a4254a914c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:12.358880 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.358820    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b0ba578-d584-4bed-92c3-e4a4254a914c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:12.360543 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.360522    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b0ba578-d584-4bed-92c3-e4a4254a914c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:12.416737 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.416671    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6557\" (UniqueName: \"kubernetes.io/projected/5b0ba578-d584-4bed-92c3-e4a4254a914c-kube-api-access-z6557\") pod \"isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:12.418481 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.418455    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:12.475126 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.475090    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:12.707759 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.707684    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"]
Oct 30 22:39:12.710116 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.710102    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:12.763461 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.763432    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-canary-promote-predictor-serving-cert\""
Oct 30 22:39:12.772635 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.772605    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"]
Oct 30 22:39:12.862453 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.862422    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/810415a1-f14a-46d7-8e48-f48ee206a9d7-cabundle-cert\") pod \"isvc-canary-promote-predictor-58d794bb6f-9g4qw\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:12.862655 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.862473    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/810415a1-f14a-46d7-8e48-f48ee206a9d7-kserve-provision-location\") pod \"isvc-canary-promote-predictor-58d794bb6f-9g4qw\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:12.862655 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.862536    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/810415a1-f14a-46d7-8e48-f48ee206a9d7-proxy-tls\") pod \"isvc-canary-promote-predictor-58d794bb6f-9g4qw\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:12.862655 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.862595    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/810415a1-f14a-46d7-8e48-f48ee206a9d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-58d794bb6f-9g4qw\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:12.862655 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.862616    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsp6v\" (UniqueName: \"kubernetes.io/projected/810415a1-f14a-46d7-8e48-f48ee206a9d7-kube-api-access-bsp6v\") pod \"isvc-canary-promote-predictor-58d794bb6f-9g4qw\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:12.897574 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.897550    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"]
Oct 30 22:39:12.899413 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:39:12.899387    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0ba578_d584_4bed_92c3_e4a4254a914c.slice/crio-34f25c12654810b1fb74e483674d04bb6619844989f9bc2f83f82d4e96c13dd5 WatchSource:0}: Error finding container 34f25c12654810b1fb74e483674d04bb6619844989f9bc2f83f82d4e96c13dd5: Status 404 returned error can't find the container with id 34f25c12654810b1fb74e483674d04bb6619844989f9bc2f83f82d4e96c13dd5
Oct 30 22:39:12.963946 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.963882    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/810415a1-f14a-46d7-8e48-f48ee206a9d7-proxy-tls\") pod \"isvc-canary-promote-predictor-58d794bb6f-9g4qw\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:12.963946 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.963930    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/810415a1-f14a-46d7-8e48-f48ee206a9d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-58d794bb6f-9g4qw\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:12.964124 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.963996    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsp6v\" (UniqueName: \"kubernetes.io/projected/810415a1-f14a-46d7-8e48-f48ee206a9d7-kube-api-access-bsp6v\") pod \"isvc-canary-promote-predictor-58d794bb6f-9g4qw\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:12.964124 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.964103    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/810415a1-f14a-46d7-8e48-f48ee206a9d7-cabundle-cert\") pod \"isvc-canary-promote-predictor-58d794bb6f-9g4qw\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:12.964252 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.964177    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/810415a1-f14a-46d7-8e48-f48ee206a9d7-kserve-provision-location\") pod \"isvc-canary-promote-predictor-58d794bb6f-9g4qw\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:12.964623 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.964560    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/810415a1-f14a-46d7-8e48-f48ee206a9d7-kserve-provision-location\") pod \"isvc-canary-promote-predictor-58d794bb6f-9g4qw\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:12.964724 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.964705    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/810415a1-f14a-46d7-8e48-f48ee206a9d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-58d794bb6f-9g4qw\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:12.964891 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.964869    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/810415a1-f14a-46d7-8e48-f48ee206a9d7-cabundle-cert\") pod \"isvc-canary-promote-predictor-58d794bb6f-9g4qw\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:12.966854 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:12.966837    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/810415a1-f14a-46d7-8e48-f48ee206a9d7-proxy-tls\") pod \"isvc-canary-promote-predictor-58d794bb6f-9g4qw\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:13.021828 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:13.021802    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsp6v\" (UniqueName: \"kubernetes.io/projected/810415a1-f14a-46d7-8e48-f48ee206a9d7-kube-api-access-bsp6v\") pod \"isvc-canary-promote-predictor-58d794bb6f-9g4qw\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:13.023677 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:13.023656    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:13.397122 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:13.397096    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"]
Oct 30 22:39:13.398752 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:39:13.398726    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810415a1_f14a_46d7_8e48_f48ee206a9d7.slice/crio-c73a40e21e61a106c364d3b3ebccc6a3b620ba5853c606d59ce9dedcf9c71480 WatchSource:0}: Error finding container c73a40e21e61a106c364d3b3ebccc6a3b620ba5853c606d59ce9dedcf9c71480: Status 404 returned error can't find the container with id c73a40e21e61a106c364d3b3ebccc6a3b620ba5853c606d59ce9dedcf9c71480
Oct 30 22:39:13.602393 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:13.602359    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" event={"ID":"810415a1-f14a-46d7-8e48-f48ee206a9d7","Type":"ContainerStarted","Data":"8259d1c0154673c96a77a92731186778f53e9ac8e841c207c4ed24e0ff5a9edb"}
Oct 30 22:39:13.602585 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:13.602425    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" event={"ID":"810415a1-f14a-46d7-8e48-f48ee206a9d7","Type":"ContainerStarted","Data":"c73a40e21e61a106c364d3b3ebccc6a3b620ba5853c606d59ce9dedcf9c71480"}
Oct 30 22:39:13.604115 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:13.604086    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" event={"ID":"5b0ba578-d584-4bed-92c3-e4a4254a914c","Type":"ContainerStarted","Data":"dfa1c2de653acdddf3dc19593ec6fd90f1589d743ce96028da174882b2d96af2"}
Oct 30 22:39:13.604234 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:13.604132    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" event={"ID":"5b0ba578-d584-4bed-92c3-e4a4254a914c","Type":"ContainerStarted","Data":"34f25c12654810b1fb74e483674d04bb6619844989f9bc2f83f82d4e96c13dd5"}
Oct 30 22:39:16.377603 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:16.377569    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Oct 30 22:39:17.623204 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:17.623119    4473 generic.go:356] "Generic (PLEG): container finished" podID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerID="8259d1c0154673c96a77a92731186778f53e9ac8e841c207c4ed24e0ff5a9edb" exitCode=0
Oct 30 22:39:17.623204 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:17.623188    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" event={"ID":"810415a1-f14a-46d7-8e48-f48ee206a9d7","Type":"ContainerDied","Data":"8259d1c0154673c96a77a92731186778f53e9ac8e841c207c4ed24e0ff5a9edb"}
Oct 30 22:39:17.624886 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:17.624867    4473 generic.go:356] "Generic (PLEG): container finished" podID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerID="dfa1c2de653acdddf3dc19593ec6fd90f1589d743ce96028da174882b2d96af2" exitCode=0
Oct 30 22:39:17.624982 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:17.624917    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" event={"ID":"5b0ba578-d584-4bed-92c3-e4a4254a914c","Type":"ContainerDied","Data":"dfa1c2de653acdddf3dc19593ec6fd90f1589d743ce96028da174882b2d96af2"}
Oct 30 22:39:18.631812 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:18.631775    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" event={"ID":"5b0ba578-d584-4bed-92c3-e4a4254a914c","Type":"ContainerStarted","Data":"a418caf46542ead4c3527aaf969dcbc38d9f5159cb6f4444d1cb6e32f89bb3f8"}
Oct 30 22:39:18.632301 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:18.631852    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" event={"ID":"5b0ba578-d584-4bed-92c3-e4a4254a914c","Type":"ContainerStarted","Data":"16d84dc5a25a243bed9728ae639d882fe65751a5c9f8dfed2d0fb8ce254dc3e7"}
Oct 30 22:39:18.632301 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:18.632187    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:18.632427 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:18.632365    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:18.633771 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:18.633735    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Oct 30 22:39:18.634066 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:18.634046    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" event={"ID":"810415a1-f14a-46d7-8e48-f48ee206a9d7","Type":"ContainerStarted","Data":"53be3f40e95307998321d54c1cd90556195a6fa99164425116ec8e9749b52d0e"}
Oct 30 22:39:18.634123 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:18.634096    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" event={"ID":"810415a1-f14a-46d7-8e48-f48ee206a9d7","Type":"ContainerStarted","Data":"f0dea49353d5a06910790fb2e6f13a14ec25e557437eb133ab96e6cae48b9cd7"}
Oct 30 22:39:18.634286 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:18.634270    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:18.753725 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:18.753673    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" podStartSLOduration=2.789679492 podStartE2EDuration="6.753657716s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.963978224s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:18.752332332 +0000 UTC m=+2776.988541673" watchObservedRunningTime="2026-10-30 22:39:18.753657716 +0000 UTC m=+2776.989867060"
Oct 30 22:39:18.872638 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:18.872577    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" podStartSLOduration=2.977059259 podStartE2EDuration="6.872561726s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.895502467s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:18.872085964 +0000 UTC m=+2777.108295310" watchObservedRunningTime="2026-10-30 22:39:18.872561726 +0000 UTC m=+2777.108771072"
Oct 30 22:39:19.639593 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:19.639550    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Oct 30 22:39:19.640045 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:19.639698    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:19.640658 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:19.640634    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Oct 30 22:39:20.643756 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:20.643721    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Oct 30 22:39:24.643588 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:24.643559    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:39:24.644104 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:24.644017    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Oct 30 22:39:25.648462 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:25.648434    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:39:25.648985 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:25.648940    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Oct 30 22:39:26.377943 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:26.377915    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:39:34.346994 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:34.346946    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"]
Oct 30 22:39:34.347432 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:34.347184    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kserve-container" containerID="cri-o://fff2bd4a3c1af7694fab03cbe30e94c9c1b5f05ba33fa51d9fcecd6d30f98f06" gracePeriod=30
Oct 30 22:39:34.347432 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:34.347220    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kube-rbac-proxy" containerID="cri-o://9e4cbdc5eb51fec8ebc30d32f581719b8dae14035e4a512422a629e0caa737c4" gracePeriod=30
Oct 30 22:39:34.575564 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:34.575519    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:34.644711 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:34.644680    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Oct 30 22:39:34.706863 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:34.706826    4473 generic.go:356] "Generic (PLEG): container finished" podID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerID="9e4cbdc5eb51fec8ebc30d32f581719b8dae14035e4a512422a629e0caa737c4" exitCode=0
Oct 30 22:39:34.707061 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:34.706904    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" event={"ID":"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9","Type":"ContainerDied","Data":"9e4cbdc5eb51fec8ebc30d32f581719b8dae14035e4a512422a629e0caa737c4"}
Oct 30 22:39:35.649638 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:35.649601    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Oct 30 22:39:36.372458 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:36.372417    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Oct 30 22:39:36.378047 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:36.378022    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Oct 30 22:39:38.594568 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.594540    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:39:38.671168 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.671131    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kube-rbac-proxy-sar-config\") pod \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") "
Oct 30 22:39:38.671354 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.671184    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kserve-provision-location\") pod \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") "
Oct 30 22:39:38.671354 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.671221    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-cabundle-cert\") pod \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") "
Oct 30 22:39:38.671354 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.671265    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kube-api-access-8x94w\" (UniqueName: \"kubernetes.io/projected/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kube-api-access-8x94w\") pod \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") "
Oct 30 22:39:38.671354 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.671348    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-proxy-tls\") pod \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\" (UID: \"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9\") "
Oct 30 22:39:38.671596 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.671564    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kube-rbac-proxy-sar-config" pod "e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" (UID: "e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:38.671653 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.671587    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kserve-provision-location" pod "e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" (UID: "e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:38.671653 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.671610    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-cabundle-cert" pod "e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" (UID: "e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:38.673672 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.673640    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kube-api-access-8x94w" pod "e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" (UID: "e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9"). InnerVolumeSpecName "kube-api-access-8x94w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:38.673767 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.673734    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-proxy-tls" pod "e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" (UID: "e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:38.726019 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.725986    4473 generic.go:356] "Generic (PLEG): container finished" podID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerID="fff2bd4a3c1af7694fab03cbe30e94c9c1b5f05ba33fa51d9fcecd6d30f98f06" exitCode=0
Oct 30 22:39:38.726156 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.726057    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" event={"ID":"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9","Type":"ContainerDied","Data":"fff2bd4a3c1af7694fab03cbe30e94c9c1b5f05ba33fa51d9fcecd6d30f98f06"}
Oct 30 22:39:38.726156 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.726107    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"
Oct 30 22:39:38.726156 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.726136    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl" event={"ID":"e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9","Type":"ContainerDied","Data":"2bb5efac216046b4325b993132712dbb8498135322661f2f6764f6593dade05c"}
Oct 30 22:39:38.726261 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.726166    4473 scope.go:122] "RemoveContainer" containerID="9e4cbdc5eb51fec8ebc30d32f581719b8dae14035e4a512422a629e0caa737c4"
Oct 30 22:39:38.734833 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.734805    4473 scope.go:122] "RemoveContainer" containerID="fff2bd4a3c1af7694fab03cbe30e94c9c1b5f05ba33fa51d9fcecd6d30f98f06"
Oct 30 22:39:38.742822 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.742803    4473 scope.go:122] "RemoveContainer" containerID="9ae3dee78211daf0bf882f7a180e7f7721def6e1e06d26d0b2528ad21ca1f18d"
Oct 30 22:39:38.750212 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.750187    4473 scope.go:122] "RemoveContainer" containerID="9e4cbdc5eb51fec8ebc30d32f581719b8dae14035e4a512422a629e0caa737c4"
Oct 30 22:39:38.750483 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:39:38.750459    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e4cbdc5eb51fec8ebc30d32f581719b8dae14035e4a512422a629e0caa737c4\": container with ID starting with 9e4cbdc5eb51fec8ebc30d32f581719b8dae14035e4a512422a629e0caa737c4 not found: ID does not exist" containerID="9e4cbdc5eb51fec8ebc30d32f581719b8dae14035e4a512422a629e0caa737c4"
Oct 30 22:39:38.750535 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.750494    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e4cbdc5eb51fec8ebc30d32f581719b8dae14035e4a512422a629e0caa737c4"} err="failed to get container status \"9e4cbdc5eb51fec8ebc30d32f581719b8dae14035e4a512422a629e0caa737c4\": rpc error: code = NotFound desc = could not find container \"9e4cbdc5eb51fec8ebc30d32f581719b8dae14035e4a512422a629e0caa737c4\": container with ID starting with 9e4cbdc5eb51fec8ebc30d32f581719b8dae14035e4a512422a629e0caa737c4 not found: ID does not exist"
Oct 30 22:39:38.750535 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.750514    4473 scope.go:122] "RemoveContainer" containerID="fff2bd4a3c1af7694fab03cbe30e94c9c1b5f05ba33fa51d9fcecd6d30f98f06"
Oct 30 22:39:38.750707 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:39:38.750694    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff2bd4a3c1af7694fab03cbe30e94c9c1b5f05ba33fa51d9fcecd6d30f98f06\": container with ID starting with fff2bd4a3c1af7694fab03cbe30e94c9c1b5f05ba33fa51d9fcecd6d30f98f06 not found: ID does not exist" containerID="fff2bd4a3c1af7694fab03cbe30e94c9c1b5f05ba33fa51d9fcecd6d30f98f06"
Oct 30 22:39:38.750747 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.750714    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff2bd4a3c1af7694fab03cbe30e94c9c1b5f05ba33fa51d9fcecd6d30f98f06"} err="failed to get container status \"fff2bd4a3c1af7694fab03cbe30e94c9c1b5f05ba33fa51d9fcecd6d30f98f06\": rpc error: code = NotFound desc = could not find container \"fff2bd4a3c1af7694fab03cbe30e94c9c1b5f05ba33fa51d9fcecd6d30f98f06\": container with ID starting with fff2bd4a3c1af7694fab03cbe30e94c9c1b5f05ba33fa51d9fcecd6d30f98f06 not found: ID does not exist"
Oct 30 22:39:38.750747 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.750730    4473 scope.go:122] "RemoveContainer" containerID="9ae3dee78211daf0bf882f7a180e7f7721def6e1e06d26d0b2528ad21ca1f18d"
Oct 30 22:39:38.750970 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:39:38.750938    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae3dee78211daf0bf882f7a180e7f7721def6e1e06d26d0b2528ad21ca1f18d\": container with ID starting with 9ae3dee78211daf0bf882f7a180e7f7721def6e1e06d26d0b2528ad21ca1f18d not found: ID does not exist" containerID="9ae3dee78211daf0bf882f7a180e7f7721def6e1e06d26d0b2528ad21ca1f18d"
Oct 30 22:39:38.751018 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.750977    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae3dee78211daf0bf882f7a180e7f7721def6e1e06d26d0b2528ad21ca1f18d"} err="failed to get container status \"9ae3dee78211daf0bf882f7a180e7f7721def6e1e06d26d0b2528ad21ca1f18d\": rpc error: code = NotFound desc = could not find container \"9ae3dee78211daf0bf882f7a180e7f7721def6e1e06d26d0b2528ad21ca1f18d\": container with ID starting with 9ae3dee78211daf0bf882f7a180e7f7721def6e1e06d26d0b2528ad21ca1f18d not found: ID does not exist"
Oct 30 22:39:38.772150 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.772122    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:38.772150 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.772152    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:38.772330 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.772166    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:38.772330 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.772178    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x94w\" (UniqueName: \"kubernetes.io/projected/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-kube-api-access-8x94w\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:38.772330 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.772191    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:38.880301 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.880269    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"]
Oct 30 22:39:38.883401 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:38.883372    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-runtime-predictor-5789f96589-lb5dl"]
Oct 30 22:39:39.886855 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:39.886809    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" path="/var/lib/kubelet/pods/e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9/volumes"
Oct 30 22:39:40.841216 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:40.841188    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:41.119988 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.119879    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:41.303476 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:39:41.303438    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:39:41.332960 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.332928    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"]
Oct 30 22:39:41.333323 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.333310    4473 cpu_manager.go:414] "containerMap: removing container" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kserve-container"
Oct 30 22:39:41.333369 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.333325    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kserve-container"
Oct 30 22:39:41.333369 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.333342    4473 cpu_manager.go:414] "containerMap: removing container" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="storage-initializer"
Oct 30 22:39:41.333369 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.333348    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="storage-initializer"
Oct 30 22:39:41.333369 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.333358    4473 cpu_manager.go:414] "containerMap: removing container" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kube-rbac-proxy"
Oct 30 22:39:41.333369 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.333365    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kube-rbac-proxy"
Oct 30 22:39:41.333522 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.333419    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kube-rbac-proxy"
Oct 30 22:39:41.333522 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.333431    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0616b1c-0d0b-449d-9ee7-e4bd87dbcef9" containerName="kserve-container"
Oct 30 22:39:41.336581 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.336561    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:41.390120 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.390035    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 22:39:41.390120 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.390046    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:39:41.390301 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.390144    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:39:41.392465 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.392448    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf78aee6-3ca7-44d5-beb2-2c291c674818-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:41.392551 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.392502    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf78aee6-3ca7-44d5-beb2-2c291c674818-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:41.392610 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.392583    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf78aee6-3ca7-44d5-beb2-2c291c674818-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:41.392669 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.392624    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf78aee6-3ca7-44d5-beb2-2c291c674818-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:41.392729 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.392698    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lsmv\" (UniqueName: \"kubernetes.io/projected/bf78aee6-3ca7-44d5-beb2-2c291c674818-kube-api-access-8lsmv\") pod \"isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:41.440598 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.440567    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Oct 30 22:39:41.440598 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.440586    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:39:41.440598 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.440567    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:39:41.440857 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.440711    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:39:41.442404 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.442382    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"]
Oct 30 22:39:41.493664 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.493635    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lsmv\" (UniqueName: \"kubernetes.io/projected/bf78aee6-3ca7-44d5-beb2-2c291c674818-kube-api-access-8lsmv\") pod \"isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:41.493849 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.493680    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf78aee6-3ca7-44d5-beb2-2c291c674818-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:41.493849 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.493770    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf78aee6-3ca7-44d5-beb2-2c291c674818-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:41.493849 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.493805    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf78aee6-3ca7-44d5-beb2-2c291c674818-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:41.493849 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.493834    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf78aee6-3ca7-44d5-beb2-2c291c674818-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:41.494302 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.494278    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf78aee6-3ca7-44d5-beb2-2c291c674818-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:41.494423 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.494401    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf78aee6-3ca7-44d5-beb2-2c291c674818-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:41.494507 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.494489    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf78aee6-3ca7-44d5-beb2-2c291c674818-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:41.496517 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.496501    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf78aee6-3ca7-44d5-beb2-2c291c674818-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:41.552575 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.552542    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lsmv\" (UniqueName: \"kubernetes.io/projected/bf78aee6-3ca7-44d5-beb2-2c291c674818-kube-api-access-8lsmv\") pod \"isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:41.649914 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:41.649880    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:42.074580 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:42.074556    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"]
Oct 30 22:39:42.077009 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:39:42.076977    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf78aee6_3ca7_44d5_beb2_2c291c674818.slice/crio-b3089034b03b2fdccec4e4812154a314f6f78803ff618da944636f1bb16e3244 WatchSource:0}: Error finding container b3089034b03b2fdccec4e4812154a314f6f78803ff618da944636f1bb16e3244: Status 404 returned error can't find the container with id b3089034b03b2fdccec4e4812154a314f6f78803ff618da944636f1bb16e3244
Oct 30 22:39:42.747739 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:42.747705    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5" event={"ID":"bf78aee6-3ca7-44d5-beb2-2c291c674818","Type":"ContainerStarted","Data":"ab9932b4fc393aa7729949da8f2a629e4e0a64ccf98a1581e912e451e6d65a55"}
Oct 30 22:39:42.748216 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:42.747789    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5" event={"ID":"bf78aee6-3ca7-44d5-beb2-2c291c674818","Type":"ContainerStarted","Data":"b3089034b03b2fdccec4e4812154a314f6f78803ff618da944636f1bb16e3244"}
Oct 30 22:39:44.644005 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:44.643961    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Oct 30 22:39:45.649222 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:45.649188    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Oct 30 22:39:46.766373 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:46.766332    4473 generic.go:356] "Generic (PLEG): container finished" podID="bf78aee6-3ca7-44d5-beb2-2c291c674818" containerID="ab9932b4fc393aa7729949da8f2a629e4e0a64ccf98a1581e912e451e6d65a55" exitCode=0
Oct 30 22:39:46.766877 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:46.766409    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5" event={"ID":"bf78aee6-3ca7-44d5-beb2-2c291c674818","Type":"ContainerDied","Data":"ab9932b4fc393aa7729949da8f2a629e4e0a64ccf98a1581e912e451e6d65a55"}
Oct 30 22:39:47.780455 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:47.780423    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5" event={"ID":"bf78aee6-3ca7-44d5-beb2-2c291c674818","Type":"ContainerStarted","Data":"adf07abece40ee19377d0fe55a80a8eab26af5a86df75eef9406201c60918145"}
Oct 30 22:39:47.780979 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:47.780524    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5" event={"ID":"bf78aee6-3ca7-44d5-beb2-2c291c674818","Type":"ContainerStarted","Data":"c8b2f29914fbb4c24e675f18b2ac56d446c1c4642aeec2be0811870b3ece651e"}
Oct 30 22:39:47.780979 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:47.780751    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:48.001371 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:48.001319    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5" podStartSLOduration=3.341707871 podStartE2EDuration="7.001305962s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.659598091s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:47.950587026 +0000 UTC m=+2806.186796394" watchObservedRunningTime="2026-10-30 22:39:48.001305962 +0000 UTC m=+2806.237515306"
Oct 30 22:39:48.784748 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:48.784718    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:54.644715 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:54.644671    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Oct 30 22:39:54.794461 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:54.794424    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:39:55.649198 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:39:55.649162    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Oct 30 22:40:04.644443 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:04.644403    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Oct 30 22:40:05.649606 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:05.649567    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Oct 30 22:40:07.179327 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:07.179268    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:11.304363 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:40:11.304323    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:40:14.644886 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:14.644796    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Oct 30 22:40:15.649252 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:15.649212    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Oct 30 22:40:24.644112 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:24.644075    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Oct 30 22:40:24.864749 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:24.864701    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5" podUID="bf78aee6-3ca7-44d5-beb2-2c291c674818" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 30 22:40:25.649871 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:25.649830    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Oct 30 22:40:26.881797 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:26.881753    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Oct 30 22:40:34.797638 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:34.797608    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:40:35.649472 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:35.649423    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Oct 30 22:40:36.882489 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:36.882445    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Oct 30 22:40:39.887168 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:39.887138    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:40:41.304282 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:40:41.304243    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:40:41.801668 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:41.801630    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"]
Oct 30 22:40:41.801882 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:41.801860    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5" podUID="bf78aee6-3ca7-44d5-beb2-2c291c674818" containerName="kserve-container" containerID="cri-o://c8b2f29914fbb4c24e675f18b2ac56d446c1c4642aeec2be0811870b3ece651e" gracePeriod=30
Oct 30 22:40:41.801979 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:41.801907    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5" podUID="bf78aee6-3ca7-44d5-beb2-2c291c674818" containerName="kube-rbac-proxy" containerID="cri-o://adf07abece40ee19377d0fe55a80a8eab26af5a86df75eef9406201c60918145" gracePeriod=30
Oct 30 22:40:41.830545 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:41.830514    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:42.021562 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:42.021534    4473 generic.go:356] "Generic (PLEG): container finished" podID="bf78aee6-3ca7-44d5-beb2-2c291c674818" containerID="adf07abece40ee19377d0fe55a80a8eab26af5a86df75eef9406201c60918145" exitCode=0
Oct 30 22:40:42.021737 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:42.021609    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5" event={"ID":"bf78aee6-3ca7-44d5-beb2-2c291c674818","Type":"ContainerDied","Data":"adf07abece40ee19377d0fe55a80a8eab26af5a86df75eef9406201c60918145"}
Oct 30 22:40:44.789031 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:44.788985    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5" podUID="bf78aee6-3ca7-44d5-beb2-2c291c674818" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Oct 30 22:40:46.882160 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:46.882129    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:40:48.296350 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:48.296315    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"]
Oct 30 22:40:48.296793 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:48.296602    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kserve-container" containerID="cri-o://f0dea49353d5a06910790fb2e6f13a14ec25e557437eb133ab96e6cae48b9cd7" gracePeriod=30
Oct 30 22:40:48.296793 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:48.296615    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kube-rbac-proxy" containerID="cri-o://53be3f40e95307998321d54c1cd90556195a6fa99164425116ec8e9749b52d0e" gracePeriod=30
Oct 30 22:40:49.055910 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.055876    4473 generic.go:356] "Generic (PLEG): container finished" podID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerID="53be3f40e95307998321d54c1cd90556195a6fa99164425116ec8e9749b52d0e" exitCode=0
Oct 30 22:40:49.056101 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.055967    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" event={"ID":"810415a1-f14a-46d7-8e48-f48ee206a9d7","Type":"ContainerDied","Data":"53be3f40e95307998321d54c1cd90556195a6fa99164425116ec8e9749b52d0e"}
Oct 30 22:40:49.252210 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.252184    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:40:49.352200 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.352125    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf78aee6-3ca7-44d5-beb2-2c291c674818-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf78aee6-3ca7-44d5-beb2-2c291c674818-kserve-provision-location\") pod \"bf78aee6-3ca7-44d5-beb2-2c291c674818\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") "
Oct 30 22:40:49.352200 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.352167    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf78aee6-3ca7-44d5-beb2-2c291c674818-kube-api-access-8lsmv\" (UniqueName: \"kubernetes.io/projected/bf78aee6-3ca7-44d5-beb2-2c291c674818-kube-api-access-8lsmv\") pod \"bf78aee6-3ca7-44d5-beb2-2c291c674818\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") "
Oct 30 22:40:49.352200 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.352195    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf78aee6-3ca7-44d5-beb2-2c291c674818-proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf78aee6-3ca7-44d5-beb2-2c291c674818-proxy-tls\") pod \"bf78aee6-3ca7-44d5-beb2-2c291c674818\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") "
Oct 30 22:40:49.352711 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.352218    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf78aee6-3ca7-44d5-beb2-2c291c674818-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf78aee6-3ca7-44d5-beb2-2c291c674818-cabundle-cert\") pod \"bf78aee6-3ca7-44d5-beb2-2c291c674818\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") "
Oct 30 22:40:49.352711 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.352446    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf78aee6-3ca7-44d5-beb2-2c291c674818-kserve-provision-location" pod "bf78aee6-3ca7-44d5-beb2-2c291c674818" (UID: "bf78aee6-3ca7-44d5-beb2-2c291c674818"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:49.352711 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.352611    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf78aee6-3ca7-44d5-beb2-2c291c674818-cabundle-cert" pod "bf78aee6-3ca7-44d5-beb2-2c291c674818" (UID: "bf78aee6-3ca7-44d5-beb2-2c291c674818"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:49.354376 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.354347    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf78aee6-3ca7-44d5-beb2-2c291c674818-proxy-tls" pod "bf78aee6-3ca7-44d5-beb2-2c291c674818" (UID: "bf78aee6-3ca7-44d5-beb2-2c291c674818"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:49.354486 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.354434    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf78aee6-3ca7-44d5-beb2-2c291c674818-kube-api-access-8lsmv" pod "bf78aee6-3ca7-44d5-beb2-2c291c674818" (UID: "bf78aee6-3ca7-44d5-beb2-2c291c674818"). InnerVolumeSpecName "kube-api-access-8lsmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:49.453003 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.452938    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf78aee6-3ca7-44d5-beb2-2c291c674818-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf78aee6-3ca7-44d5-beb2-2c291c674818-kube-rbac-proxy-sar-config\") pod \"bf78aee6-3ca7-44d5-beb2-2c291c674818\" (UID: \"bf78aee6-3ca7-44d5-beb2-2c291c674818\") "
Oct 30 22:40:49.453171 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.453121    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lsmv\" (UniqueName: \"kubernetes.io/projected/bf78aee6-3ca7-44d5-beb2-2c291c674818-kube-api-access-8lsmv\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:49.453171 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.453132    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf78aee6-3ca7-44d5-beb2-2c291c674818-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:49.453171 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.453143    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf78aee6-3ca7-44d5-beb2-2c291c674818-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:49.453171 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.453152    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf78aee6-3ca7-44d5-beb2-2c291c674818-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:49.453328 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.453261    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf78aee6-3ca7-44d5-beb2-2c291c674818-kube-rbac-proxy-sar-config" pod "bf78aee6-3ca7-44d5-beb2-2c291c674818" (UID: "bf78aee6-3ca7-44d5-beb2-2c291c674818"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:49.554088 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.554057    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf78aee6-3ca7-44d5-beb2-2c291c674818-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:49.882055 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:49.882014    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Oct 30 22:40:50.062284 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:50.062249    4473 generic.go:356] "Generic (PLEG): container finished" podID="bf78aee6-3ca7-44d5-beb2-2c291c674818" containerID="c8b2f29914fbb4c24e675f18b2ac56d446c1c4642aeec2be0811870b3ece651e" exitCode=0
Oct 30 22:40:50.062458 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:50.062293    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5" event={"ID":"bf78aee6-3ca7-44d5-beb2-2c291c674818","Type":"ContainerDied","Data":"c8b2f29914fbb4c24e675f18b2ac56d446c1c4642aeec2be0811870b3ece651e"}
Oct 30 22:40:50.062458 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:50.062313    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"
Oct 30 22:40:50.062458 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:50.062341    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5" event={"ID":"bf78aee6-3ca7-44d5-beb2-2c291c674818","Type":"ContainerDied","Data":"b3089034b03b2fdccec4e4812154a314f6f78803ff618da944636f1bb16e3244"}
Oct 30 22:40:50.062458 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:50.062368    4473 scope.go:122] "RemoveContainer" containerID="adf07abece40ee19377d0fe55a80a8eab26af5a86df75eef9406201c60918145"
Oct 30 22:40:50.070817 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:50.070794    4473 scope.go:122] "RemoveContainer" containerID="c8b2f29914fbb4c24e675f18b2ac56d446c1c4642aeec2be0811870b3ece651e"
Oct 30 22:40:50.078534 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:50.078506    4473 scope.go:122] "RemoveContainer" containerID="ab9932b4fc393aa7729949da8f2a629e4e0a64ccf98a1581e912e451e6d65a55"
Oct 30 22:40:50.085815 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:50.085790    4473 scope.go:122] "RemoveContainer" containerID="adf07abece40ee19377d0fe55a80a8eab26af5a86df75eef9406201c60918145"
Oct 30 22:40:50.086090 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:40:50.086071    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adf07abece40ee19377d0fe55a80a8eab26af5a86df75eef9406201c60918145\": container with ID starting with adf07abece40ee19377d0fe55a80a8eab26af5a86df75eef9406201c60918145 not found: ID does not exist" containerID="adf07abece40ee19377d0fe55a80a8eab26af5a86df75eef9406201c60918145"
Oct 30 22:40:50.086135 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:50.086102    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf07abece40ee19377d0fe55a80a8eab26af5a86df75eef9406201c60918145"} err="failed to get container status \"adf07abece40ee19377d0fe55a80a8eab26af5a86df75eef9406201c60918145\": rpc error: code = NotFound desc = could not find container \"adf07abece40ee19377d0fe55a80a8eab26af5a86df75eef9406201c60918145\": container with ID starting with adf07abece40ee19377d0fe55a80a8eab26af5a86df75eef9406201c60918145 not found: ID does not exist"
Oct 30 22:40:50.086135 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:50.086120    4473 scope.go:122] "RemoveContainer" containerID="c8b2f29914fbb4c24e675f18b2ac56d446c1c4642aeec2be0811870b3ece651e"
Oct 30 22:40:50.086339 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:40:50.086323    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b2f29914fbb4c24e675f18b2ac56d446c1c4642aeec2be0811870b3ece651e\": container with ID starting with c8b2f29914fbb4c24e675f18b2ac56d446c1c4642aeec2be0811870b3ece651e not found: ID does not exist" containerID="c8b2f29914fbb4c24e675f18b2ac56d446c1c4642aeec2be0811870b3ece651e"
Oct 30 22:40:50.086388 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:50.086346    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b2f29914fbb4c24e675f18b2ac56d446c1c4642aeec2be0811870b3ece651e"} err="failed to get container status \"c8b2f29914fbb4c24e675f18b2ac56d446c1c4642aeec2be0811870b3ece651e\": rpc error: code = NotFound desc = could not find container \"c8b2f29914fbb4c24e675f18b2ac56d446c1c4642aeec2be0811870b3ece651e\": container with ID starting with c8b2f29914fbb4c24e675f18b2ac56d446c1c4642aeec2be0811870b3ece651e not found: ID does not exist"
Oct 30 22:40:50.086388 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:50.086360    4473 scope.go:122] "RemoveContainer" containerID="ab9932b4fc393aa7729949da8f2a629e4e0a64ccf98a1581e912e451e6d65a55"
Oct 30 22:40:50.086591 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:40:50.086574    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9932b4fc393aa7729949da8f2a629e4e0a64ccf98a1581e912e451e6d65a55\": container with ID starting with ab9932b4fc393aa7729949da8f2a629e4e0a64ccf98a1581e912e451e6d65a55 not found: ID does not exist" containerID="ab9932b4fc393aa7729949da8f2a629e4e0a64ccf98a1581e912e451e6d65a55"
Oct 30 22:40:50.086638 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:50.086593    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9932b4fc393aa7729949da8f2a629e4e0a64ccf98a1581e912e451e6d65a55"} err="failed to get container status \"ab9932b4fc393aa7729949da8f2a629e4e0a64ccf98a1581e912e451e6d65a55\": rpc error: code = NotFound desc = could not find container \"ab9932b4fc393aa7729949da8f2a629e4e0a64ccf98a1581e912e451e6d65a55\": container with ID starting with ab9932b4fc393aa7729949da8f2a629e4e0a64ccf98a1581e912e451e6d65a55 not found: ID does not exist"
Oct 30 22:40:50.228023 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:50.227988    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"]
Oct 30 22:40:50.278397 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:50.278365    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-runtime-predictor-55df7d566-vqht5"]
Oct 30 22:40:50.644112 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:50.644019    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Oct 30 22:40:51.887458 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:51.887428    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf78aee6-3ca7-44d5-beb2-2c291c674818" path="/var/lib/kubelet/pods/bf78aee6-3ca7-44d5-beb2-2c291c674818/volumes"
Oct 30 22:40:52.053318 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:52.053284    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:53.088330 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.088298    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:53.257711 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.257679    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:53.444396 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.444372    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:40:53.482365 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.482331    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/810415a1-f14a-46d7-8e48-f48ee206a9d7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/810415a1-f14a-46d7-8e48-f48ee206a9d7-kserve-provision-location\") pod \"810415a1-f14a-46d7-8e48-f48ee206a9d7\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") "
Oct 30 22:40:53.482533 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.482401    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/810415a1-f14a-46d7-8e48-f48ee206a9d7-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/810415a1-f14a-46d7-8e48-f48ee206a9d7-cabundle-cert\") pod \"810415a1-f14a-46d7-8e48-f48ee206a9d7\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") "
Oct 30 22:40:53.482533 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.482426    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/810415a1-f14a-46d7-8e48-f48ee206a9d7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/810415a1-f14a-46d7-8e48-f48ee206a9d7-kube-rbac-proxy-sar-config\") pod \"810415a1-f14a-46d7-8e48-f48ee206a9d7\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") "
Oct 30 22:40:53.482533 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.482447    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/810415a1-f14a-46d7-8e48-f48ee206a9d7-proxy-tls\" (UniqueName: \"kubernetes.io/secret/810415a1-f14a-46d7-8e48-f48ee206a9d7-proxy-tls\") pod \"810415a1-f14a-46d7-8e48-f48ee206a9d7\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") "
Oct 30 22:40:53.482533 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.482480    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/810415a1-f14a-46d7-8e48-f48ee206a9d7-kube-api-access-bsp6v\" (UniqueName: \"kubernetes.io/projected/810415a1-f14a-46d7-8e48-f48ee206a9d7-kube-api-access-bsp6v\") pod \"810415a1-f14a-46d7-8e48-f48ee206a9d7\" (UID: \"810415a1-f14a-46d7-8e48-f48ee206a9d7\") "
Oct 30 22:40:53.482715 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.482632    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/810415a1-f14a-46d7-8e48-f48ee206a9d7-kserve-provision-location" pod "810415a1-f14a-46d7-8e48-f48ee206a9d7" (UID: "810415a1-f14a-46d7-8e48-f48ee206a9d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:53.482837 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.482815    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/810415a1-f14a-46d7-8e48-f48ee206a9d7-kube-rbac-proxy-sar-config" pod "810415a1-f14a-46d7-8e48-f48ee206a9d7" (UID: "810415a1-f14a-46d7-8e48-f48ee206a9d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:53.482837 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.482823    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/810415a1-f14a-46d7-8e48-f48ee206a9d7-cabundle-cert" pod "810415a1-f14a-46d7-8e48-f48ee206a9d7" (UID: "810415a1-f14a-46d7-8e48-f48ee206a9d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:53.484697 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.484676    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/810415a1-f14a-46d7-8e48-f48ee206a9d7-kube-api-access-bsp6v" pod "810415a1-f14a-46d7-8e48-f48ee206a9d7" (UID: "810415a1-f14a-46d7-8e48-f48ee206a9d7"). InnerVolumeSpecName "kube-api-access-bsp6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:53.484697 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.484688    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/810415a1-f14a-46d7-8e48-f48ee206a9d7-proxy-tls" pod "810415a1-f14a-46d7-8e48-f48ee206a9d7" (UID: "810415a1-f14a-46d7-8e48-f48ee206a9d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:53.554728 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.554681    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"]
Oct 30 22:40:53.555126 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.555108    4473 cpu_manager.go:414] "containerMap: removing container" podUID="bf78aee6-3ca7-44d5-beb2-2c291c674818" containerName="storage-initializer"
Oct 30 22:40:53.555223 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.555127    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf78aee6-3ca7-44d5-beb2-2c291c674818" containerName="storage-initializer"
Oct 30 22:40:53.555223 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.555160    4473 cpu_manager.go:414] "containerMap: removing container" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kserve-container"
Oct 30 22:40:53.555223 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.555169    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kserve-container"
Oct 30 22:40:53.555223 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.555183    4473 cpu_manager.go:414] "containerMap: removing container" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="storage-initializer"
Oct 30 22:40:53.555223 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.555191    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="storage-initializer"
Oct 30 22:40:53.555223 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.555205    4473 cpu_manager.go:414] "containerMap: removing container" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kube-rbac-proxy"
Oct 30 22:40:53.555223 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.555213    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kube-rbac-proxy"
Oct 30 22:40:53.555223 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.555224    4473 cpu_manager.go:414] "containerMap: removing container" podUID="bf78aee6-3ca7-44d5-beb2-2c291c674818" containerName="kserve-container"
Oct 30 22:40:53.555620 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.555231    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf78aee6-3ca7-44d5-beb2-2c291c674818" containerName="kserve-container"
Oct 30 22:40:53.555620 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.555252    4473 cpu_manager.go:414] "containerMap: removing container" podUID="bf78aee6-3ca7-44d5-beb2-2c291c674818" containerName="kube-rbac-proxy"
Oct 30 22:40:53.555620 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.555260    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf78aee6-3ca7-44d5-beb2-2c291c674818" containerName="kube-rbac-proxy"
Oct 30 22:40:53.555620 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.555337    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kserve-container"
Oct 30 22:40:53.555620 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.555350    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerName="kube-rbac-proxy"
Oct 30 22:40:53.555620 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.555360    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf78aee6-3ca7-44d5-beb2-2c291c674818" containerName="kserve-container"
Oct 30 22:40:53.555620 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.555372    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf78aee6-3ca7-44d5-beb2-2c291c674818" containerName="kube-rbac-proxy"
Oct 30 22:40:53.558773 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.558751    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:40:53.583415 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.583384    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55805fbf-7914-485e-80bb-13f437d6e667-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:40:53.583540 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.583420    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55805fbf-7914-485e-80bb-13f437d6e667-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:40:53.583540 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.583439    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqg7n\" (UniqueName: \"kubernetes.io/projected/55805fbf-7914-485e-80bb-13f437d6e667-kube-api-access-qqg7n\") pod \"isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:40:53.583540 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.583486    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55805fbf-7914-485e-80bb-13f437d6e667-proxy-tls\") pod \"isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:40:53.583645 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.583607    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55805fbf-7914-485e-80bb-13f437d6e667-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:40:53.583679 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.583652    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/810415a1-f14a-46d7-8e48-f48ee206a9d7-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:53.583679 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.583664    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/810415a1-f14a-46d7-8e48-f48ee206a9d7-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:53.583740 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.583691    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/810415a1-f14a-46d7-8e48-f48ee206a9d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:53.583740 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.583703    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/810415a1-f14a-46d7-8e48-f48ee206a9d7-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:53.583740 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.583713    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsp6v\" (UniqueName: \"kubernetes.io/projected/810415a1-f14a-46d7-8e48-f48ee206a9d7-kube-api-access-bsp6v\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:53.616082 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.615989    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"seaweedfs-s3-creds\""
Oct 30 22:40:53.665806 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.665772    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:40:53.665806 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.665796    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:40:53.666062 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.666038    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Oct 30 22:40:53.666299 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.666284    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Oct 30 22:40:53.669282 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.669264    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"]
Oct 30 22:40:53.684893 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.684872    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55805fbf-7914-485e-80bb-13f437d6e667-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:40:53.684992 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.684902    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55805fbf-7914-485e-80bb-13f437d6e667-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:40:53.684992 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.684923    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55805fbf-7914-485e-80bb-13f437d6e667-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:40:53.685080 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.685040    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqg7n\" (UniqueName: \"kubernetes.io/projected/55805fbf-7914-485e-80bb-13f437d6e667-kube-api-access-qqg7n\") pod \"isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:40:53.685142 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.685076    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55805fbf-7914-485e-80bb-13f437d6e667-proxy-tls\") pod \"isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:40:53.685343 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.685323    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55805fbf-7914-485e-80bb-13f437d6e667-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:40:53.685594 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.685578    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55805fbf-7914-485e-80bb-13f437d6e667-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:40:53.687573 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.687544    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55805fbf-7914-485e-80bb-13f437d6e667-proxy-tls\") pod \"isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:40:53.715669 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.715636    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:40:53.715838 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.715822    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:40:53.726576 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.726548    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55805fbf-7914-485e-80bb-13f437d6e667-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:40:53.745523 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.745493    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqg7n\" (UniqueName: \"kubernetes.io/projected/55805fbf-7914-485e-80bb-13f437d6e667-kube-api-access-qqg7n\") pod \"isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") " pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:40:53.875973 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:53.875858    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:40:54.088625 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:54.088590    4473 generic.go:356] "Generic (PLEG): container finished" podID="810415a1-f14a-46d7-8e48-f48ee206a9d7" containerID="f0dea49353d5a06910790fb2e6f13a14ec25e557437eb133ab96e6cae48b9cd7" exitCode=0
Oct 30 22:40:54.089142 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:54.088646    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"
Oct 30 22:40:54.089142 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:54.088660    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" event={"ID":"810415a1-f14a-46d7-8e48-f48ee206a9d7","Type":"ContainerDied","Data":"f0dea49353d5a06910790fb2e6f13a14ec25e557437eb133ab96e6cae48b9cd7"}
Oct 30 22:40:54.089142 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:54.088737    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw" event={"ID":"810415a1-f14a-46d7-8e48-f48ee206a9d7","Type":"ContainerDied","Data":"c73a40e21e61a106c364d3b3ebccc6a3b620ba5853c606d59ce9dedcf9c71480"}
Oct 30 22:40:54.089142 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:54.088781    4473 scope.go:122] "RemoveContainer" containerID="53be3f40e95307998321d54c1cd90556195a6fa99164425116ec8e9749b52d0e"
Oct 30 22:40:54.096827 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:54.096811    4473 scope.go:122] "RemoveContainer" containerID="f0dea49353d5a06910790fb2e6f13a14ec25e557437eb133ab96e6cae48b9cd7"
Oct 30 22:40:54.104167 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:54.104143    4473 scope.go:122] "RemoveContainer" containerID="8259d1c0154673c96a77a92731186778f53e9ac8e841c207c4ed24e0ff5a9edb"
Oct 30 22:40:54.111632 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:54.111615    4473 scope.go:122] "RemoveContainer" containerID="53be3f40e95307998321d54c1cd90556195a6fa99164425116ec8e9749b52d0e"
Oct 30 22:40:54.111876 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:40:54.111861    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53be3f40e95307998321d54c1cd90556195a6fa99164425116ec8e9749b52d0e\": container with ID starting with 53be3f40e95307998321d54c1cd90556195a6fa99164425116ec8e9749b52d0e not found: ID does not exist" containerID="53be3f40e95307998321d54c1cd90556195a6fa99164425116ec8e9749b52d0e"
Oct 30 22:40:54.111922 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:54.111885    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53be3f40e95307998321d54c1cd90556195a6fa99164425116ec8e9749b52d0e"} err="failed to get container status \"53be3f40e95307998321d54c1cd90556195a6fa99164425116ec8e9749b52d0e\": rpc error: code = NotFound desc = could not find container \"53be3f40e95307998321d54c1cd90556195a6fa99164425116ec8e9749b52d0e\": container with ID starting with 53be3f40e95307998321d54c1cd90556195a6fa99164425116ec8e9749b52d0e not found: ID does not exist"
Oct 30 22:40:54.111922 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:54.111901    4473 scope.go:122] "RemoveContainer" containerID="f0dea49353d5a06910790fb2e6f13a14ec25e557437eb133ab96e6cae48b9cd7"
Oct 30 22:40:54.112139 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:40:54.112123    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0dea49353d5a06910790fb2e6f13a14ec25e557437eb133ab96e6cae48b9cd7\": container with ID starting with f0dea49353d5a06910790fb2e6f13a14ec25e557437eb133ab96e6cae48b9cd7 not found: ID does not exist" containerID="f0dea49353d5a06910790fb2e6f13a14ec25e557437eb133ab96e6cae48b9cd7"
Oct 30 22:40:54.112183 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:54.112146    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0dea49353d5a06910790fb2e6f13a14ec25e557437eb133ab96e6cae48b9cd7"} err="failed to get container status \"f0dea49353d5a06910790fb2e6f13a14ec25e557437eb133ab96e6cae48b9cd7\": rpc error: code = NotFound desc = could not find container \"f0dea49353d5a06910790fb2e6f13a14ec25e557437eb133ab96e6cae48b9cd7\": container with ID starting with f0dea49353d5a06910790fb2e6f13a14ec25e557437eb133ab96e6cae48b9cd7 not found: ID does not exist"
Oct 30 22:40:54.112183 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:54.112161    4473 scope.go:122] "RemoveContainer" containerID="8259d1c0154673c96a77a92731186778f53e9ac8e841c207c4ed24e0ff5a9edb"
Oct 30 22:40:54.112340 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:40:54.112324    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8259d1c0154673c96a77a92731186778f53e9ac8e841c207c4ed24e0ff5a9edb\": container with ID starting with 8259d1c0154673c96a77a92731186778f53e9ac8e841c207c4ed24e0ff5a9edb not found: ID does not exist" containerID="8259d1c0154673c96a77a92731186778f53e9ac8e841c207c4ed24e0ff5a9edb"
Oct 30 22:40:54.112384 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:54.112344    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8259d1c0154673c96a77a92731186778f53e9ac8e841c207c4ed24e0ff5a9edb"} err="failed to get container status \"8259d1c0154673c96a77a92731186778f53e9ac8e841c207c4ed24e0ff5a9edb\": rpc error: code = NotFound desc = could not find container \"8259d1c0154673c96a77a92731186778f53e9ac8e841c207c4ed24e0ff5a9edb\": container with ID starting with 8259d1c0154673c96a77a92731186778f53e9ac8e841c207c4ed24e0ff5a9edb not found: ID does not exist"
Oct 30 22:40:54.256007 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:54.255973    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"]
Oct 30 22:40:54.259758 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:54.259736    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-promote-predictor-58d794bb6f-9g4qw"]
Oct 30 22:40:54.298661 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:54.298621    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"]
Oct 30 22:40:54.308664 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:40:54.308635    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55805fbf_7914_485e_80bb_13f437d6e667.slice/crio-63f7cf406ccc6dce665c449c9e02095846734a9632fa00483331b095a314cce5 WatchSource:0}: Error finding container 63f7cf406ccc6dce665c449c9e02095846734a9632fa00483331b095a314cce5: Status 404 returned error can't find the container with id 63f7cf406ccc6dce665c449c9e02095846734a9632fa00483331b095a314cce5
Oct 30 22:40:55.095054 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:55.095017    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" event={"ID":"55805fbf-7914-485e-80bb-13f437d6e667","Type":"ContainerStarted","Data":"d6b252e047fe36c76e75dab68ad8b406f2a654bb1a93273de4a6fc97b5438aba"}
Oct 30 22:40:55.095579 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:55.095069    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" event={"ID":"55805fbf-7914-485e-80bb-13f437d6e667","Type":"ContainerStarted","Data":"63f7cf406ccc6dce665c449c9e02095846734a9632fa00483331b095a314cce5"}
Oct 30 22:40:55.885825 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:55.885793    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="810415a1-f14a-46d7-8e48-f48ee206a9d7" path="/var/lib/kubelet/pods/810415a1-f14a-46d7-8e48-f48ee206a9d7/volumes"
Oct 30 22:40:59.113124 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:59.113032    4473 generic.go:356] "Generic (PLEG): container finished" podID="55805fbf-7914-485e-80bb-13f437d6e667" containerID="d6b252e047fe36c76e75dab68ad8b406f2a654bb1a93273de4a6fc97b5438aba" exitCode=0
Oct 30 22:40:59.113537 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:40:59.113117    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" event={"ID":"55805fbf-7914-485e-80bb-13f437d6e667","Type":"ContainerDied","Data":"d6b252e047fe36c76e75dab68ad8b406f2a654bb1a93273de4a6fc97b5438aba"}
Oct 30 22:41:00.119566 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:41:00.119534    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" event={"ID":"55805fbf-7914-485e-80bb-13f437d6e667","Type":"ContainerStarted","Data":"3ac7ea91c8fabc22420ec719635e7e0e34126ab4e2f5c053d39318b85a07f9e4"}
Oct 30 22:41:00.120027 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:41:00.119609    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" event={"ID":"55805fbf-7914-485e-80bb-13f437d6e667","Type":"ContainerStarted","Data":"24202f9214edb3c96690337255ad526e2eabee7bb6bbdd8d381fc6847cac684b"}
Oct 30 22:41:00.120027 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:41:00.119875    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:41:00.120027 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:41:00.120017    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:41:00.121299 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:41:00.121275    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Oct 30 22:41:00.292892 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:41:00.292835    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" podStartSLOduration=2.998750796 podStartE2EDuration="7.292818625s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.294067829s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:00.292128554 +0000 UTC m=+2878.528337901" watchObservedRunningTime="2026-10-30 22:41:00.292818625 +0000 UTC m=+2878.529028045"
Oct 30 22:41:01.125108 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:41:01.125065    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Oct 30 22:41:06.130072 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:41:06.130040    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:41:06.130586 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:41:06.130563    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Oct 30 22:41:07.511274 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:41:07.511233    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:11.303642 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:41:11.303607    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:41:16.130570 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:41:16.130533    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Oct 30 22:41:26.130591 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:41:26.130552    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Oct 30 22:41:36.131095 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:41:36.131057    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Oct 30 22:41:41.303550 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:41:41.303511    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:41:46.130943 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:41:46.130862    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Oct 30 22:41:56.131017 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:41:56.130978    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Oct 30 22:42:06.131979 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:06.131919    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:42:07.842097 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:07.842062    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:11.303707 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:42:11.303670    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:42:13.346602 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:13.346572    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"]
Oct 30 22:42:13.347028 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:13.346806    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kserve-container" containerID="cri-o://24202f9214edb3c96690337255ad526e2eabee7bb6bbdd8d381fc6847cac684b" gracePeriod=30
Oct 30 22:42:13.347028 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:13.346820    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kube-rbac-proxy" containerID="cri-o://3ac7ea91c8fabc22420ec719635e7e0e34126ab4e2f5c053d39318b85a07f9e4" gracePeriod=30
Oct 30 22:42:13.378359 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:13.378330    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:14.423793 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:14.423763    4473 generic.go:356] "Generic (PLEG): container finished" podID="55805fbf-7914-485e-80bb-13f437d6e667" containerID="3ac7ea91c8fabc22420ec719635e7e0e34126ab4e2f5c053d39318b85a07f9e4" exitCode=0
Oct 30 22:42:14.424223 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:14.423839    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" event={"ID":"55805fbf-7914-485e-80bb-13f437d6e667","Type":"ContainerDied","Data":"3ac7ea91c8fabc22420ec719635e7e0e34126ab4e2f5c053d39318b85a07f9e4"}
Oct 30 22:42:16.125397 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:16.125359    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Oct 30 22:42:16.130736 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:16.130711    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Oct 30 22:42:17.587151 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:17.587127    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:42:17.666598 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:17.666566    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55805fbf-7914-485e-80bb-13f437d6e667-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55805fbf-7914-485e-80bb-13f437d6e667-kserve-provision-location\") pod \"55805fbf-7914-485e-80bb-13f437d6e667\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") "
Oct 30 22:42:17.666745 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:17.666631    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55805fbf-7914-485e-80bb-13f437d6e667-kube-api-access-qqg7n\" (UniqueName: \"kubernetes.io/projected/55805fbf-7914-485e-80bb-13f437d6e667-kube-api-access-qqg7n\") pod \"55805fbf-7914-485e-80bb-13f437d6e667\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") "
Oct 30 22:42:17.666745 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:17.666696    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55805fbf-7914-485e-80bb-13f437d6e667-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55805fbf-7914-485e-80bb-13f437d6e667-kube-rbac-proxy-sar-config\") pod \"55805fbf-7914-485e-80bb-13f437d6e667\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") "
Oct 30 22:42:17.666745 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:17.666722    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55805fbf-7914-485e-80bb-13f437d6e667-proxy-tls\" (UniqueName: \"kubernetes.io/secret/55805fbf-7914-485e-80bb-13f437d6e667-proxy-tls\") pod \"55805fbf-7914-485e-80bb-13f437d6e667\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") "
Oct 30 22:42:17.666745 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:17.666743    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55805fbf-7914-485e-80bb-13f437d6e667-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55805fbf-7914-485e-80bb-13f437d6e667-cabundle-cert\") pod \"55805fbf-7914-485e-80bb-13f437d6e667\" (UID: \"55805fbf-7914-485e-80bb-13f437d6e667\") "
Oct 30 22:42:17.667056 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:17.666938    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55805fbf-7914-485e-80bb-13f437d6e667-kserve-provision-location" pod "55805fbf-7914-485e-80bb-13f437d6e667" (UID: "55805fbf-7914-485e-80bb-13f437d6e667"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:17.667182 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:17.667119    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55805fbf-7914-485e-80bb-13f437d6e667-kube-rbac-proxy-sar-config" pod "55805fbf-7914-485e-80bb-13f437d6e667" (UID: "55805fbf-7914-485e-80bb-13f437d6e667"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:17.667229 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:17.667183    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55805fbf-7914-485e-80bb-13f437d6e667-cabundle-cert" pod "55805fbf-7914-485e-80bb-13f437d6e667" (UID: "55805fbf-7914-485e-80bb-13f437d6e667"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:17.668876 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:17.668853    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55805fbf-7914-485e-80bb-13f437d6e667-kube-api-access-qqg7n" pod "55805fbf-7914-485e-80bb-13f437d6e667" (UID: "55805fbf-7914-485e-80bb-13f437d6e667"). InnerVolumeSpecName "kube-api-access-qqg7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:17.669057 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:17.668891    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55805fbf-7914-485e-80bb-13f437d6e667-proxy-tls" pod "55805fbf-7914-485e-80bb-13f437d6e667" (UID: "55805fbf-7914-485e-80bb-13f437d6e667"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:17.767987 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:17.767945    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqg7n\" (UniqueName: \"kubernetes.io/projected/55805fbf-7914-485e-80bb-13f437d6e667-kube-api-access-qqg7n\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:17.767987 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:17.767985    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55805fbf-7914-485e-80bb-13f437d6e667-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:17.768178 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:17.767996    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55805fbf-7914-485e-80bb-13f437d6e667-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:17.768178 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:17.768005    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55805fbf-7914-485e-80bb-13f437d6e667-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:17.768178 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:17.768014    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55805fbf-7914-485e-80bb-13f437d6e667-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:18.442872 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:18.442831    4473 generic.go:356] "Generic (PLEG): container finished" podID="55805fbf-7914-485e-80bb-13f437d6e667" containerID="24202f9214edb3c96690337255ad526e2eabee7bb6bbdd8d381fc6847cac684b" exitCode=0
Oct 30 22:42:18.443099 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:18.442892    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"
Oct 30 22:42:18.443099 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:18.442922    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" event={"ID":"55805fbf-7914-485e-80bb-13f437d6e667","Type":"ContainerDied","Data":"24202f9214edb3c96690337255ad526e2eabee7bb6bbdd8d381fc6847cac684b"}
Oct 30 22:42:18.443099 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:18.443005    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt" event={"ID":"55805fbf-7914-485e-80bb-13f437d6e667","Type":"ContainerDied","Data":"63f7cf406ccc6dce665c449c9e02095846734a9632fa00483331b095a314cce5"}
Oct 30 22:42:18.443099 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:18.443044    4473 scope.go:122] "RemoveContainer" containerID="3ac7ea91c8fabc22420ec719635e7e0e34126ab4e2f5c053d39318b85a07f9e4"
Oct 30 22:42:18.451134 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:18.451115    4473 scope.go:122] "RemoveContainer" containerID="24202f9214edb3c96690337255ad526e2eabee7bb6bbdd8d381fc6847cac684b"
Oct 30 22:42:18.458563 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:18.458539    4473 scope.go:122] "RemoveContainer" containerID="d6b252e047fe36c76e75dab68ad8b406f2a654bb1a93273de4a6fc97b5438aba"
Oct 30 22:42:18.465817 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:18.465800    4473 scope.go:122] "RemoveContainer" containerID="3ac7ea91c8fabc22420ec719635e7e0e34126ab4e2f5c053d39318b85a07f9e4"
Oct 30 22:42:18.466088 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:42:18.466069    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac7ea91c8fabc22420ec719635e7e0e34126ab4e2f5c053d39318b85a07f9e4\": container with ID starting with 3ac7ea91c8fabc22420ec719635e7e0e34126ab4e2f5c053d39318b85a07f9e4 not found: ID does not exist" containerID="3ac7ea91c8fabc22420ec719635e7e0e34126ab4e2f5c053d39318b85a07f9e4"
Oct 30 22:42:18.466146 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:18.466098    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac7ea91c8fabc22420ec719635e7e0e34126ab4e2f5c053d39318b85a07f9e4"} err="failed to get container status \"3ac7ea91c8fabc22420ec719635e7e0e34126ab4e2f5c053d39318b85a07f9e4\": rpc error: code = NotFound desc = could not find container \"3ac7ea91c8fabc22420ec719635e7e0e34126ab4e2f5c053d39318b85a07f9e4\": container with ID starting with 3ac7ea91c8fabc22420ec719635e7e0e34126ab4e2f5c053d39318b85a07f9e4 not found: ID does not exist"
Oct 30 22:42:18.466146 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:18.466115    4473 scope.go:122] "RemoveContainer" containerID="24202f9214edb3c96690337255ad526e2eabee7bb6bbdd8d381fc6847cac684b"
Oct 30 22:42:18.466349 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:42:18.466331    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24202f9214edb3c96690337255ad526e2eabee7bb6bbdd8d381fc6847cac684b\": container with ID starting with 24202f9214edb3c96690337255ad526e2eabee7bb6bbdd8d381fc6847cac684b not found: ID does not exist" containerID="24202f9214edb3c96690337255ad526e2eabee7bb6bbdd8d381fc6847cac684b"
Oct 30 22:42:18.466401 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:18.466357    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24202f9214edb3c96690337255ad526e2eabee7bb6bbdd8d381fc6847cac684b"} err="failed to get container status \"24202f9214edb3c96690337255ad526e2eabee7bb6bbdd8d381fc6847cac684b\": rpc error: code = NotFound desc = could not find container \"24202f9214edb3c96690337255ad526e2eabee7bb6bbdd8d381fc6847cac684b\": container with ID starting with 24202f9214edb3c96690337255ad526e2eabee7bb6bbdd8d381fc6847cac684b not found: ID does not exist"
Oct 30 22:42:18.466401 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:18.466372    4473 scope.go:122] "RemoveContainer" containerID="d6b252e047fe36c76e75dab68ad8b406f2a654bb1a93273de4a6fc97b5438aba"
Oct 30 22:42:18.466598 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:42:18.466583    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b252e047fe36c76e75dab68ad8b406f2a654bb1a93273de4a6fc97b5438aba\": container with ID starting with d6b252e047fe36c76e75dab68ad8b406f2a654bb1a93273de4a6fc97b5438aba not found: ID does not exist" containerID="d6b252e047fe36c76e75dab68ad8b406f2a654bb1a93273de4a6fc97b5438aba"
Oct 30 22:42:18.466646 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:18.466602    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b252e047fe36c76e75dab68ad8b406f2a654bb1a93273de4a6fc97b5438aba"} err="failed to get container status \"d6b252e047fe36c76e75dab68ad8b406f2a654bb1a93273de4a6fc97b5438aba\": rpc error: code = NotFound desc = could not find container \"d6b252e047fe36c76e75dab68ad8b406f2a654bb1a93273de4a6fc97b5438aba\": container with ID starting with d6b252e047fe36c76e75dab68ad8b406f2a654bb1a93273de4a6fc97b5438aba not found: ID does not exist"
Oct 30 22:42:18.660833 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:18.660798    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"]
Oct 30 22:42:18.711783 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:18.711706    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-xgboost-v2-predictor-7cbd48bdc7-5tggt"]
Oct 30 22:42:19.885754 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:19.885722    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55805fbf-7914-485e-80bb-13f437d6e667" path="/var/lib/kubelet/pods/55805fbf-7914-485e-80bb-13f437d6e667/volumes"
Oct 30 22:42:22.224011 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.223944    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:22.456097 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.456027    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:22.600857 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.600761    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"]
Oct 30 22:42:22.601206 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.601188    4473 cpu_manager.go:414] "containerMap: removing container" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kube-rbac-proxy"
Oct 30 22:42:22.601300 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.601208    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kube-rbac-proxy"
Oct 30 22:42:22.601300 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.601231    4473 cpu_manager.go:414] "containerMap: removing container" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="storage-initializer"
Oct 30 22:42:22.601300 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.601238    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="storage-initializer"
Oct 30 22:42:22.601300 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.601258    4473 cpu_manager.go:414] "containerMap: removing container" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kserve-container"
Oct 30 22:42:22.601300 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.601266    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kserve-container"
Oct 30 22:42:22.601554 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.601361    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kube-rbac-proxy"
Oct 30 22:42:22.601554 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.601376    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="55805fbf-7914-485e-80bb-13f437d6e667" containerName="kserve-container"
Oct 30 22:42:22.606147 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.606125    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:42:22.659566 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.659527    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"storage-config\""
Oct 30 22:42:22.659566 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.659540    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:42:22.659566 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.659544    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:42:22.659795 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.659555    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Oct 30 22:42:22.709532 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.709498    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b049552e-94fe-43a0-b308-e6fc75565d50-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6d8958c898-7x2fw\" (UID: \"b049552e-94fe-43a0-b308-e6fc75565d50\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:42:22.709710 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.709542    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b049552e-94fe-43a0-b308-e6fc75565d50-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6d8958c898-7x2fw\" (UID: \"b049552e-94fe-43a0-b308-e6fc75565d50\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:42:22.709710 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.709637    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nfsg\" (UniqueName: \"kubernetes.io/projected/b049552e-94fe-43a0-b308-e6fc75565d50-kube-api-access-7nfsg\") pod \"isvc-sklearn-s3-predictor-6d8958c898-7x2fw\" (UID: \"b049552e-94fe-43a0-b308-e6fc75565d50\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:42:22.709808 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.709740    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b049552e-94fe-43a0-b308-e6fc75565d50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6d8958c898-7x2fw\" (UID: \"b049552e-94fe-43a0-b308-e6fc75565d50\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:42:22.710285 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.710262    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Oct 30 22:42:22.710405 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.710305    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:42:22.711773 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.711751    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"]
Oct 30 22:42:22.810633 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.810595    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b049552e-94fe-43a0-b308-e6fc75565d50-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6d8958c898-7x2fw\" (UID: \"b049552e-94fe-43a0-b308-e6fc75565d50\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:42:22.810821 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.810640    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b049552e-94fe-43a0-b308-e6fc75565d50-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6d8958c898-7x2fw\" (UID: \"b049552e-94fe-43a0-b308-e6fc75565d50\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:42:22.810821 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.810671    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nfsg\" (UniqueName: \"kubernetes.io/projected/b049552e-94fe-43a0-b308-e6fc75565d50-kube-api-access-7nfsg\") pod \"isvc-sklearn-s3-predictor-6d8958c898-7x2fw\" (UID: \"b049552e-94fe-43a0-b308-e6fc75565d50\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:42:22.810821 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.810713    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b049552e-94fe-43a0-b308-e6fc75565d50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6d8958c898-7x2fw\" (UID: \"b049552e-94fe-43a0-b308-e6fc75565d50\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:42:22.811071 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.811053    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b049552e-94fe-43a0-b308-e6fc75565d50-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6d8958c898-7x2fw\" (UID: \"b049552e-94fe-43a0-b308-e6fc75565d50\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:42:22.811399 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.811380    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b049552e-94fe-43a0-b308-e6fc75565d50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6d8958c898-7x2fw\" (UID: \"b049552e-94fe-43a0-b308-e6fc75565d50\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:42:22.813206 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.813191    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b049552e-94fe-43a0-b308-e6fc75565d50-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6d8958c898-7x2fw\" (UID: \"b049552e-94fe-43a0-b308-e6fc75565d50\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:42:22.869092 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.869022    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nfsg\" (UniqueName: \"kubernetes.io/projected/b049552e-94fe-43a0-b308-e6fc75565d50-kube-api-access-7nfsg\") pod \"isvc-sklearn-s3-predictor-6d8958c898-7x2fw\" (UID: \"b049552e-94fe-43a0-b308-e6fc75565d50\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:42:22.925320 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:22.925276    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:42:23.348976 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:23.348935    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"]
Oct 30 22:42:23.351529 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:42:23.351491    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb049552e_94fe_43a0_b308_e6fc75565d50.slice/crio-a3273d23953efca187d81fd1ec1a394869f53bac28bcb28ea5f4f9aceb59b326 WatchSource:0}: Error finding container a3273d23953efca187d81fd1ec1a394869f53bac28bcb28ea5f4f9aceb59b326: Status 404 returned error can't find the container with id a3273d23953efca187d81fd1ec1a394869f53bac28bcb28ea5f4f9aceb59b326
Oct 30 22:42:23.465514 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:23.465473    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" event={"ID":"b049552e-94fe-43a0-b308-e6fc75565d50","Type":"ContainerStarted","Data":"b05abaa7d786d3734589f6beed2d7d1119934fcc207f114f4970d393c2b695b0"}
Oct 30 22:42:23.465681 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:23.465537    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" event={"ID":"b049552e-94fe-43a0-b308-e6fc75565d50","Type":"ContainerStarted","Data":"a3273d23953efca187d81fd1ec1a394869f53bac28bcb28ea5f4f9aceb59b326"}
Oct 30 22:42:23.866776 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:23.866728    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"]
Oct 30 22:42:23.867036 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:23.867004    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kserve-container" containerID="cri-o://16d84dc5a25a243bed9728ae639d882fe65751a5c9f8dfed2d0fb8ce254dc3e7" gracePeriod=30
Oct 30 22:42:23.867181 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:23.867047    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kube-rbac-proxy" containerID="cri-o://a418caf46542ead4c3527aaf969dcbc38d9f5159cb6f4444d1cb6e32f89bb3f8" gracePeriod=30
Oct 30 22:42:23.926860 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:23.926832    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:24.471693 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:24.471663    4473 generic.go:356] "Generic (PLEG): container finished" podID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerID="a418caf46542ead4c3527aaf969dcbc38d9f5159cb6f4444d1cb6e32f89bb3f8" exitCode=0
Oct 30 22:42:24.472079 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:24.471759    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" event={"ID":"5b0ba578-d584-4bed-92c3-e4a4254a914c","Type":"ContainerDied","Data":"a418caf46542ead4c3527aaf969dcbc38d9f5159cb6f4444d1cb6e32f89bb3f8"}
Oct 30 22:42:24.473321 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:24.473301    4473 generic.go:356] "Generic (PLEG): container finished" podID="b049552e-94fe-43a0-b308-e6fc75565d50" containerID="b05abaa7d786d3734589f6beed2d7d1119934fcc207f114f4970d393c2b695b0" exitCode=0
Oct 30 22:42:24.473433 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:24.473369    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" event={"ID":"b049552e-94fe-43a0-b308-e6fc75565d50","Type":"ContainerDied","Data":"b05abaa7d786d3734589f6beed2d7d1119934fcc207f114f4970d393c2b695b0"}
Oct 30 22:42:24.640361 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:24.640320    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Oct 30 22:42:25.479654 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:25.479618    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" event={"ID":"b049552e-94fe-43a0-b308-e6fc75565d50","Type":"ContainerStarted","Data":"a7d495c0072eac4c5b47ac1766e68951ffd14a0c56cb3afc0685de5b94924e5f"}
Oct 30 22:42:25.480089 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:25.479685    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" event={"ID":"b049552e-94fe-43a0-b308-e6fc75565d50","Type":"ContainerStarted","Data":"5173af0de539633995e06c8df3bd29e9c101db2ae379c821ddbb82c56b187231"}
Oct 30 22:42:25.480089 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:25.479718    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:42:25.601020 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:25.600942    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" podStartSLOduration=2.581954321 podStartE2EDuration="3.600927045s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.018972724s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:25.600503931 +0000 UTC m=+2963.836713276" watchObservedRunningTime="2026-10-30 22:42:25.600927045 +0000 UTC m=+2963.837136389"
Oct 30 22:42:26.484506 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:26.484468    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:42:26.486038 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:26.486010    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Oct 30 22:42:26.882284 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:26.882178    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Oct 30 22:42:27.488112 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:27.488072    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Oct 30 22:42:28.913868 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:28.913843    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:42:28.960154 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:28.960078    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0ba578-d584-4bed-92c3-e4a4254a914c-kube-api-access-z6557\" (UniqueName: \"kubernetes.io/projected/5b0ba578-d584-4bed-92c3-e4a4254a914c-kube-api-access-z6557\") pod \"5b0ba578-d584-4bed-92c3-e4a4254a914c\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") "
Oct 30 22:42:28.960154 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:28.960113    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b0ba578-d584-4bed-92c3-e4a4254a914c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b0ba578-d584-4bed-92c3-e4a4254a914c-kube-rbac-proxy-sar-config\") pod \"5b0ba578-d584-4bed-92c3-e4a4254a914c\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") "
Oct 30 22:42:28.960154 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:28.960143    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0ba578-d584-4bed-92c3-e4a4254a914c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b0ba578-d584-4bed-92c3-e4a4254a914c-kserve-provision-location\") pod \"5b0ba578-d584-4bed-92c3-e4a4254a914c\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") "
Oct 30 22:42:28.960412 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:28.960182    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b0ba578-d584-4bed-92c3-e4a4254a914c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b0ba578-d584-4bed-92c3-e4a4254a914c-proxy-tls\") pod \"5b0ba578-d584-4bed-92c3-e4a4254a914c\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") "
Oct 30 22:42:28.960412 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:28.960239    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b0ba578-d584-4bed-92c3-e4a4254a914c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b0ba578-d584-4bed-92c3-e4a4254a914c-cabundle-cert\") pod \"5b0ba578-d584-4bed-92c3-e4a4254a914c\" (UID: \"5b0ba578-d584-4bed-92c3-e4a4254a914c\") "
Oct 30 22:42:28.960562 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:28.960525    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0ba578-d584-4bed-92c3-e4a4254a914c-kserve-provision-location" pod "5b0ba578-d584-4bed-92c3-e4a4254a914c" (UID: "5b0ba578-d584-4bed-92c3-e4a4254a914c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:28.960625 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:28.960569    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0ba578-d584-4bed-92c3-e4a4254a914c-kube-rbac-proxy-sar-config" pod "5b0ba578-d584-4bed-92c3-e4a4254a914c" (UID: "5b0ba578-d584-4bed-92c3-e4a4254a914c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:28.960625 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:28.960613    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0ba578-d584-4bed-92c3-e4a4254a914c-cabundle-cert" pod "5b0ba578-d584-4bed-92c3-e4a4254a914c" (UID: "5b0ba578-d584-4bed-92c3-e4a4254a914c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:28.962409 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:28.962388    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0ba578-d584-4bed-92c3-e4a4254a914c-proxy-tls" pod "5b0ba578-d584-4bed-92c3-e4a4254a914c" (UID: "5b0ba578-d584-4bed-92c3-e4a4254a914c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:28.962481 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:28.962449    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0ba578-d584-4bed-92c3-e4a4254a914c-kube-api-access-z6557" pod "5b0ba578-d584-4bed-92c3-e4a4254a914c" (UID: "5b0ba578-d584-4bed-92c3-e4a4254a914c"). InnerVolumeSpecName "kube-api-access-z6557". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:29.060888 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.060856    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6557\" (UniqueName: \"kubernetes.io/projected/5b0ba578-d584-4bed-92c3-e4a4254a914c-kube-api-access-z6557\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:29.060888 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.060884    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b0ba578-d584-4bed-92c3-e4a4254a914c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:29.060888 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.060894    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b0ba578-d584-4bed-92c3-e4a4254a914c-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:29.061151 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.060905    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b0ba578-d584-4bed-92c3-e4a4254a914c-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:29.061151 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.060915    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b0ba578-d584-4bed-92c3-e4a4254a914c-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:29.498020 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.497980    4473 generic.go:356] "Generic (PLEG): container finished" podID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerID="16d84dc5a25a243bed9728ae639d882fe65751a5c9f8dfed2d0fb8ce254dc3e7" exitCode=0
Oct 30 22:42:29.498192 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.498041    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" event={"ID":"5b0ba578-d584-4bed-92c3-e4a4254a914c","Type":"ContainerDied","Data":"16d84dc5a25a243bed9728ae639d882fe65751a5c9f8dfed2d0fb8ce254dc3e7"}
Oct 30 22:42:29.498192 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.498056    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"
Oct 30 22:42:29.498192 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.498127    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn" event={"ID":"5b0ba578-d584-4bed-92c3-e4a4254a914c","Type":"ContainerDied","Data":"34f25c12654810b1fb74e483674d04bb6619844989f9bc2f83f82d4e96c13dd5"}
Oct 30 22:42:29.498192 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.498154    4473 scope.go:122] "RemoveContainer" containerID="a418caf46542ead4c3527aaf969dcbc38d9f5159cb6f4444d1cb6e32f89bb3f8"
Oct 30 22:42:29.508946 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.508779    4473 scope.go:122] "RemoveContainer" containerID="16d84dc5a25a243bed9728ae639d882fe65751a5c9f8dfed2d0fb8ce254dc3e7"
Oct 30 22:42:29.517056 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.517039    4473 scope.go:122] "RemoveContainer" containerID="dfa1c2de653acdddf3dc19593ec6fd90f1589d743ce96028da174882b2d96af2"
Oct 30 22:42:29.524372 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.524352    4473 scope.go:122] "RemoveContainer" containerID="a418caf46542ead4c3527aaf969dcbc38d9f5159cb6f4444d1cb6e32f89bb3f8"
Oct 30 22:42:29.524638 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:42:29.524620    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a418caf46542ead4c3527aaf969dcbc38d9f5159cb6f4444d1cb6e32f89bb3f8\": container with ID starting with a418caf46542ead4c3527aaf969dcbc38d9f5159cb6f4444d1cb6e32f89bb3f8 not found: ID does not exist" containerID="a418caf46542ead4c3527aaf969dcbc38d9f5159cb6f4444d1cb6e32f89bb3f8"
Oct 30 22:42:29.524703 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.524646    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a418caf46542ead4c3527aaf969dcbc38d9f5159cb6f4444d1cb6e32f89bb3f8"} err="failed to get container status \"a418caf46542ead4c3527aaf969dcbc38d9f5159cb6f4444d1cb6e32f89bb3f8\": rpc error: code = NotFound desc = could not find container \"a418caf46542ead4c3527aaf969dcbc38d9f5159cb6f4444d1cb6e32f89bb3f8\": container with ID starting with a418caf46542ead4c3527aaf969dcbc38d9f5159cb6f4444d1cb6e32f89bb3f8 not found: ID does not exist"
Oct 30 22:42:29.524703 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.524663    4473 scope.go:122] "RemoveContainer" containerID="16d84dc5a25a243bed9728ae639d882fe65751a5c9f8dfed2d0fb8ce254dc3e7"
Oct 30 22:42:29.525081 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:42:29.525056    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d84dc5a25a243bed9728ae639d882fe65751a5c9f8dfed2d0fb8ce254dc3e7\": container with ID starting with 16d84dc5a25a243bed9728ae639d882fe65751a5c9f8dfed2d0fb8ce254dc3e7 not found: ID does not exist" containerID="16d84dc5a25a243bed9728ae639d882fe65751a5c9f8dfed2d0fb8ce254dc3e7"
Oct 30 22:42:29.525130 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.525093    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d84dc5a25a243bed9728ae639d882fe65751a5c9f8dfed2d0fb8ce254dc3e7"} err="failed to get container status \"16d84dc5a25a243bed9728ae639d882fe65751a5c9f8dfed2d0fb8ce254dc3e7\": rpc error: code = NotFound desc = could not find container \"16d84dc5a25a243bed9728ae639d882fe65751a5c9f8dfed2d0fb8ce254dc3e7\": container with ID starting with 16d84dc5a25a243bed9728ae639d882fe65751a5c9f8dfed2d0fb8ce254dc3e7 not found: ID does not exist"
Oct 30 22:42:29.525130 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.525119    4473 scope.go:122] "RemoveContainer" containerID="dfa1c2de653acdddf3dc19593ec6fd90f1589d743ce96028da174882b2d96af2"
Oct 30 22:42:29.525362 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:42:29.525348    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa1c2de653acdddf3dc19593ec6fd90f1589d743ce96028da174882b2d96af2\": container with ID starting with dfa1c2de653acdddf3dc19593ec6fd90f1589d743ce96028da174882b2d96af2 not found: ID does not exist" containerID="dfa1c2de653acdddf3dc19593ec6fd90f1589d743ce96028da174882b2d96af2"
Oct 30 22:42:29.525413 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.525366    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa1c2de653acdddf3dc19593ec6fd90f1589d743ce96028da174882b2d96af2"} err="failed to get container status \"dfa1c2de653acdddf3dc19593ec6fd90f1589d743ce96028da174882b2d96af2\": rpc error: code = NotFound desc = could not find container \"dfa1c2de653acdddf3dc19593ec6fd90f1589d743ce96028da174882b2d96af2\": container with ID starting with dfa1c2de653acdddf3dc19593ec6fd90f1589d743ce96028da174882b2d96af2 not found: ID does not exist"
Oct 30 22:42:29.665643 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.665603    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"]
Oct 30 22:42:29.669600 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.669567    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-promote-v2-predictor-6d897ccb86-v2bdn"]
Oct 30 22:42:29.887031 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:29.886940    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" path="/var/lib/kubelet/pods/5b0ba578-d584-4bed-92c3-e4a4254a914c/volumes"
Oct 30 22:42:32.493150 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:32.493121    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:42:32.493652 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:32.493626    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Oct 30 22:42:32.534097 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:32.534052    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:32.701540 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:32.701500    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:32.969902 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:32.969874    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"]
Oct 30 22:42:32.970230 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:32.970215    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kube-rbac-proxy"
Oct 30 22:42:32.970295 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:32.970231    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kube-rbac-proxy"
Oct 30 22:42:32.970295 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:32.970251    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kserve-container"
Oct 30 22:42:32.970295 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:32.970256    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kserve-container"
Oct 30 22:42:32.970295 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:32.970274    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="storage-initializer"
Oct 30 22:42:32.970295 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:32.970280    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="storage-initializer"
Oct 30 22:42:32.970538 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:32.970328    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kube-rbac-proxy"
Oct 30 22:42:32.970538 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:32.970338    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0ba578-d584-4bed-92c3-e4a4254a914c" containerName="kserve-container"
Oct 30 22:42:32.973450 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:32.973432    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:33.021779 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.021746    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:33.028840 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.028812    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"seaweedfs-s3-creds\""
Oct 30 22:42:33.028990 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.028858    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"default-dockercfg-4v4s8\""
Oct 30 22:42:33.029117 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.029098    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Oct 30 22:42:33.029176 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.029104    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"openshift-service-ca.crt\""
Oct 30 22:42:33.079230 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.079197    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Oct 30 22:42:33.079230 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.079220    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"kube-root-ca.crt\""
Oct 30 22:42:33.079508 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.079490    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:42:33.080920 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.080902    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"]
Oct 30 22:42:33.100330 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.100301    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f38a5e6-8045-40c9-b61d-545e80d440e7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:33.100462 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.100346    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f38a5e6-8045-40c9-b61d-545e80d440e7-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:33.100462 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.100368    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g697c\" (UniqueName: \"kubernetes.io/projected/5f38a5e6-8045-40c9-b61d-545e80d440e7-kube-api-access-g697c\") pod \"isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:33.100564 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.100504    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f38a5e6-8045-40c9-b61d-545e80d440e7-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:33.100564 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.100534    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f38a5e6-8045-40c9-b61d-545e80d440e7-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:33.201551 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.201520    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f38a5e6-8045-40c9-b61d-545e80d440e7-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:33.201551 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.201556    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f38a5e6-8045-40c9-b61d-545e80d440e7-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:33.201810 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.201584    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f38a5e6-8045-40c9-b61d-545e80d440e7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:33.201810 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.201625    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f38a5e6-8045-40c9-b61d-545e80d440e7-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:33.201810 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.201655    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g697c\" (UniqueName: \"kubernetes.io/projected/5f38a5e6-8045-40c9-b61d-545e80d440e7-kube-api-access-g697c\") pod \"isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:33.202087 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.202057    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f38a5e6-8045-40c9-b61d-545e80d440e7-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:33.202323 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.202303    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f38a5e6-8045-40c9-b61d-545e80d440e7-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:33.202395 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.202377    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f38a5e6-8045-40c9-b61d-545e80d440e7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:33.204278 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.204255    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f38a5e6-8045-40c9-b61d-545e80d440e7-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:33.260603 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.260526    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g697c\" (UniqueName: \"kubernetes.io/projected/5f38a5e6-8045-40c9-b61d-545e80d440e7-kube-api-access-g697c\") pod \"isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:33.290254 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.290212    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:33.303113 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.303081    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:33.499937 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.499907    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"]
Oct 30 22:42:33.504482 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.504466    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:33.558777 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.558748    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-canary-rollback-predictor-serving-cert\""
Oct 30 22:42:33.564924 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.564897    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"]
Oct 30 22:42:33.604643 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.604610    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nktfz\" (UniqueName: \"kubernetes.io/projected/0ec25e63-211c-4d69-865e-8f7229640eca-kube-api-access-nktfz\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-pxs69\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:33.604810 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.604727    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ec25e63-211c-4d69-865e-8f7229640eca-proxy-tls\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-pxs69\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:33.604810 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.604758    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ec25e63-211c-4d69-865e-8f7229640eca-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-pxs69\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:33.604880 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.604810    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ec25e63-211c-4d69-865e-8f7229640eca-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-pxs69\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:33.604880 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.604833    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ec25e63-211c-4d69-865e-8f7229640eca-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-pxs69\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:33.689520 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.689493    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"]
Oct 30 22:42:33.692171 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:42:33.692142    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f38a5e6_8045_40c9_b61d_545e80d440e7.slice/crio-6a6b2ab55f62c065d86c02fd0f052abc134d2d397bb23e2ead5831677f0e28e2 WatchSource:0}: Error finding container 6a6b2ab55f62c065d86c02fd0f052abc134d2d397bb23e2ead5831677f0e28e2: Status 404 returned error can't find the container with id 6a6b2ab55f62c065d86c02fd0f052abc134d2d397bb23e2ead5831677f0e28e2
Oct 30 22:42:33.706274 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.706246    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nktfz\" (UniqueName: \"kubernetes.io/projected/0ec25e63-211c-4d69-865e-8f7229640eca-kube-api-access-nktfz\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-pxs69\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:33.706394 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.706377    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ec25e63-211c-4d69-865e-8f7229640eca-proxy-tls\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-pxs69\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:33.706439 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.706411    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ec25e63-211c-4d69-865e-8f7229640eca-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-pxs69\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:33.706489 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.706440    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ec25e63-211c-4d69-865e-8f7229640eca-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-pxs69\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:33.706489 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.706471    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ec25e63-211c-4d69-865e-8f7229640eca-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-pxs69\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:33.707053 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.707032    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ec25e63-211c-4d69-865e-8f7229640eca-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-pxs69\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:33.707248 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.707230    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ec25e63-211c-4d69-865e-8f7229640eca-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-pxs69\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:33.707604 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.707575    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ec25e63-211c-4d69-865e-8f7229640eca-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-pxs69\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:33.709245 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.709225    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ec25e63-211c-4d69-865e-8f7229640eca-proxy-tls\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-pxs69\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:33.765496 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.765429    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nktfz\" (UniqueName: \"kubernetes.io/projected/0ec25e63-211c-4d69-865e-8f7229640eca-kube-api-access-nktfz\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-pxs69\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:33.821667 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:33.821631    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:34.202115 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:34.202093    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"]
Oct 30 22:42:34.203808 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:42:34.203779    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec25e63_211c_4d69_865e_8f7229640eca.slice/crio-812740e71152bcd96ec65434926ec652fea84b0e1eeb6b72e5a684cb040b7a4d WatchSource:0}: Error finding container 812740e71152bcd96ec65434926ec652fea84b0e1eeb6b72e5a684cb040b7a4d: Status 404 returned error can't find the container with id 812740e71152bcd96ec65434926ec652fea84b0e1eeb6b72e5a684cb040b7a4d
Oct 30 22:42:34.523556 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:34.523458    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" event={"ID":"0ec25e63-211c-4d69-865e-8f7229640eca","Type":"ContainerStarted","Data":"6dc08bf8804f5bd45db84087771d227d48f4966f2f976c465cf0fcf5e480960f"}
Oct 30 22:42:34.523556 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:34.523530    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" event={"ID":"0ec25e63-211c-4d69-865e-8f7229640eca","Type":"ContainerStarted","Data":"812740e71152bcd96ec65434926ec652fea84b0e1eeb6b72e5a684cb040b7a4d"}
Oct 30 22:42:34.525239 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:34.525201    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" event={"ID":"5f38a5e6-8045-40c9-b61d-545e80d440e7","Type":"ContainerStarted","Data":"cfed2820bd0cf34e31d09c24b851b45f15fcd4cc6df27591186e8588f87b8167"}
Oct 30 22:42:34.525330 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:34.525271    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" event={"ID":"5f38a5e6-8045-40c9-b61d-545e80d440e7","Type":"ContainerStarted","Data":"6a6b2ab55f62c065d86c02fd0f052abc134d2d397bb23e2ead5831677f0e28e2"}
Oct 30 22:42:38.544042 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:38.544006    4473 generic.go:356] "Generic (PLEG): container finished" podID="0ec25e63-211c-4d69-865e-8f7229640eca" containerID="6dc08bf8804f5bd45db84087771d227d48f4966f2f976c465cf0fcf5e480960f" exitCode=0
Oct 30 22:42:38.544469 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:38.544079    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" event={"ID":"0ec25e63-211c-4d69-865e-8f7229640eca","Type":"ContainerDied","Data":"6dc08bf8804f5bd45db84087771d227d48f4966f2f976c465cf0fcf5e480960f"}
Oct 30 22:42:38.545737 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:38.545712    4473 generic.go:356] "Generic (PLEG): container finished" podID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerID="cfed2820bd0cf34e31d09c24b851b45f15fcd4cc6df27591186e8588f87b8167" exitCode=0
Oct 30 22:42:38.545828 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:38.545735    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" event={"ID":"5f38a5e6-8045-40c9-b61d-545e80d440e7","Type":"ContainerDied","Data":"cfed2820bd0cf34e31d09c24b851b45f15fcd4cc6df27591186e8588f87b8167"}
Oct 30 22:42:39.552098 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:39.552064    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" event={"ID":"5f38a5e6-8045-40c9-b61d-545e80d440e7","Type":"ContainerStarted","Data":"351d7fb62ac249658bebd3c6cb207fe8be7c3b967c52d341576e05f211b79796"}
Oct 30 22:42:39.552717 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:39.552166    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" event={"ID":"5f38a5e6-8045-40c9-b61d-545e80d440e7","Type":"ContainerStarted","Data":"cd01803dbf487ce362aeaf14879e56edd402fa7da62ce5cb80fc9c17d461863d"}
Oct 30 22:42:39.552717 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:39.552408    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:39.554281 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:39.554256    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" event={"ID":"0ec25e63-211c-4d69-865e-8f7229640eca","Type":"ContainerStarted","Data":"ed003e209e83e8d40535f529ea2eb14e51ab8046b1b3b2630a5aa3e82656aa95"}
Oct 30 22:42:39.554404 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:39.554305    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" event={"ID":"0ec25e63-211c-4d69-865e-8f7229640eca","Type":"ContainerStarted","Data":"52f8619552c13b7cdd1a5dd6a51162f7f90050113ccfe128ddb322e8416f308e"}
Oct 30 22:42:39.554629 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:39.554612    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:39.554700 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:39.554657    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:39.555904 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:39.555881    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Oct 30 22:42:39.680153 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:39.680099    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" podStartSLOduration=3.687780602 podStartE2EDuration="7.680085632s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.99230503s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:39.678647028 +0000 UTC m=+2977.914856373" watchObservedRunningTime="2026-10-30 22:42:39.680085632 +0000 UTC m=+2977.916294973"
Oct 30 22:42:39.797723 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:39.797668    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" podStartSLOduration=2.976805414 podStartE2EDuration="6.797652531s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.820847117s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:39.796780657 +0000 UTC m=+2978.032990003" watchObservedRunningTime="2026-10-30 22:42:39.797652531 +0000 UTC m=+2978.033861875"
Oct 30 22:42:40.558933 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:40.558890    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Oct 30 22:42:40.559373 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:40.558942    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:40.560211 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:40.560190    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Oct 30 22:42:41.303712 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:42:41.303676    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:42:41.562946 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:41.562859    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Oct 30 22:42:42.494248 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:42.494212    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Oct 30 22:42:45.563521 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:45.563492    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:42:45.564058 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:45.564021    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Oct 30 22:42:46.568918 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:46.568887    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:42:46.569494 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:46.569465    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Oct 30 22:42:52.493813 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:52.493772    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Oct 30 22:42:55.564185 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:55.564142    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Oct 30 22:42:56.570358 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:42:56.570314    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Oct 30 22:43:02.161692 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:02.161668    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:43:02.168671 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:02.168651    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:43:02.493710 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:02.493621    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Oct 30 22:43:05.564927 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:05.564878    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Oct 30 22:43:06.569448 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:06.569400    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Oct 30 22:43:08.221811 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:08.221775    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:11.303505 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:43:11.303458    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:43:12.494636 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:12.494550    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Oct 30 22:43:15.564194 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:15.564148    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Oct 30 22:43:16.569746 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:16.569655    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Oct 30 22:43:22.494027 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:22.493990    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Oct 30 22:43:25.563978 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:25.563912    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Oct 30 22:43:26.570067 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:26.570022    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Oct 30 22:43:32.493556 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:32.493518    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Oct 30 22:43:35.564643 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:35.564595    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Oct 30 22:43:36.569745 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:36.569702    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Oct 30 22:43:41.303848 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:43:41.303812    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:43:41.889706 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:41.889660    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Oct 30 22:43:45.564483 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:45.564438    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Oct 30 22:43:46.570121 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:46.570071    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Oct 30 22:43:51.237628 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:51.237593    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:51.883900 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:51.883864    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Oct 30 22:43:51.890114 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:51.890095    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:43:55.564200 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:55.564151    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Oct 30 22:43:59.885248 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:43:59.885210    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:01.886827 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:01.886800    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:44:03.656633 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:03.656588    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"]
Oct 30 22:44:03.657150 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:03.656905    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kserve-container" containerID="cri-o://5173af0de539633995e06c8df3bd29e9c101db2ae379c821ddbb82c56b187231" gracePeriod=30
Oct 30 22:44:03.657150 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:03.656923    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kube-rbac-proxy" containerID="cri-o://a7d495c0072eac4c5b47ac1766e68951ffd14a0c56cb3afc0685de5b94924e5f" gracePeriod=30
Oct 30 22:44:03.667061 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:03.667036    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:03.923516 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:03.923435    4473 generic.go:356] "Generic (PLEG): container finished" podID="b049552e-94fe-43a0-b308-e6fc75565d50" containerID="a7d495c0072eac4c5b47ac1766e68951ffd14a0c56cb3afc0685de5b94924e5f" exitCode=0
Oct 30 22:44:03.923662 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:03.923514    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" event={"ID":"b049552e-94fe-43a0-b308-e6fc75565d50","Type":"ContainerDied","Data":"a7d495c0072eac4c5b47ac1766e68951ffd14a0c56cb3afc0685de5b94924e5f"}
Oct 30 22:44:03.955320 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:03.955286    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"]
Oct 30 22:44:03.955550 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:03.955528    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kserve-container" containerID="cri-o://52f8619552c13b7cdd1a5dd6a51162f7f90050113ccfe128ddb322e8416f308e" gracePeriod=30
Oct 30 22:44:03.955615 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:03.955587    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kube-rbac-proxy" containerID="cri-o://ed003e209e83e8d40535f529ea2eb14e51ab8046b1b3b2630a5aa3e82656aa95" gracePeriod=30
Oct 30 22:44:03.956130 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:03.956112    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:44:04.930143 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:04.930111    4473 generic.go:356] "Generic (PLEG): container finished" podID="0ec25e63-211c-4d69-865e-8f7229640eca" containerID="ed003e209e83e8d40535f529ea2eb14e51ab8046b1b3b2630a5aa3e82656aa95" exitCode=0
Oct 30 22:44:04.930531 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:04.930186    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" event={"ID":"0ec25e63-211c-4d69-865e-8f7229640eca","Type":"ContainerDied","Data":"ed003e209e83e8d40535f529ea2eb14e51ab8046b1b3b2630a5aa3e82656aa95"}
Oct 30 22:44:05.559295 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:05.559253    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Oct 30 22:44:07.488784 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:07.488742    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Oct 30 22:44:08.565702 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:08.565667    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:08.947597 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:08.947564    4473 generic.go:356] "Generic (PLEG): container finished" podID="b049552e-94fe-43a0-b308-e6fc75565d50" containerID="5173af0de539633995e06c8df3bd29e9c101db2ae379c821ddbb82c56b187231" exitCode=0
Oct 30 22:44:08.947747 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:08.947636    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" event={"ID":"b049552e-94fe-43a0-b308-e6fc75565d50","Type":"ContainerDied","Data":"5173af0de539633995e06c8df3bd29e9c101db2ae379c821ddbb82c56b187231"}
Oct 30 22:44:08.978103 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:08.978080    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:44:08.992337 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:08.992294    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b049552e-94fe-43a0-b308-e6fc75565d50-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b049552e-94fe-43a0-b308-e6fc75565d50-kube-rbac-proxy-sar-config\") pod \"b049552e-94fe-43a0-b308-e6fc75565d50\" (UID: \"b049552e-94fe-43a0-b308-e6fc75565d50\") "
Oct 30 22:44:08.992530 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:08.992343    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b049552e-94fe-43a0-b308-e6fc75565d50-proxy-tls\" (UniqueName: \"kubernetes.io/secret/b049552e-94fe-43a0-b308-e6fc75565d50-proxy-tls\") pod \"b049552e-94fe-43a0-b308-e6fc75565d50\" (UID: \"b049552e-94fe-43a0-b308-e6fc75565d50\") "
Oct 30 22:44:08.992530 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:08.992385    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b049552e-94fe-43a0-b308-e6fc75565d50-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b049552e-94fe-43a0-b308-e6fc75565d50-kserve-provision-location\") pod \"b049552e-94fe-43a0-b308-e6fc75565d50\" (UID: \"b049552e-94fe-43a0-b308-e6fc75565d50\") "
Oct 30 22:44:08.992530 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:08.992422    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b049552e-94fe-43a0-b308-e6fc75565d50-kube-api-access-7nfsg\" (UniqueName: \"kubernetes.io/projected/b049552e-94fe-43a0-b308-e6fc75565d50-kube-api-access-7nfsg\") pod \"b049552e-94fe-43a0-b308-e6fc75565d50\" (UID: \"b049552e-94fe-43a0-b308-e6fc75565d50\") "
Oct 30 22:44:08.992703 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:08.992675    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b049552e-94fe-43a0-b308-e6fc75565d50-kube-rbac-proxy-sar-config" pod "b049552e-94fe-43a0-b308-e6fc75565d50" (UID: "b049552e-94fe-43a0-b308-e6fc75565d50"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:08.992763 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:08.992720    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b049552e-94fe-43a0-b308-e6fc75565d50-kserve-provision-location" pod "b049552e-94fe-43a0-b308-e6fc75565d50" (UID: "b049552e-94fe-43a0-b308-e6fc75565d50"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:08.994737 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:08.994700    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b049552e-94fe-43a0-b308-e6fc75565d50-proxy-tls" pod "b049552e-94fe-43a0-b308-e6fc75565d50" (UID: "b049552e-94fe-43a0-b308-e6fc75565d50"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:44:08.994852 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:08.994751    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b049552e-94fe-43a0-b308-e6fc75565d50-kube-api-access-7nfsg" pod "b049552e-94fe-43a0-b308-e6fc75565d50" (UID: "b049552e-94fe-43a0-b308-e6fc75565d50"). InnerVolumeSpecName "kube-api-access-7nfsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:09.088964 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.088926    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:44:09.092934 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.092914    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec25e63-211c-4d69-865e-8f7229640eca-kube-api-access-nktfz\" (UniqueName: \"kubernetes.io/projected/0ec25e63-211c-4d69-865e-8f7229640eca-kube-api-access-nktfz\") pod \"0ec25e63-211c-4d69-865e-8f7229640eca\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") "
Oct 30 22:44:09.093011 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.092994    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec25e63-211c-4d69-865e-8f7229640eca-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ec25e63-211c-4d69-865e-8f7229640eca-kube-rbac-proxy-sar-config\") pod \"0ec25e63-211c-4d69-865e-8f7229640eca\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") "
Oct 30 22:44:09.093048 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.093017    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec25e63-211c-4d69-865e-8f7229640eca-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ec25e63-211c-4d69-865e-8f7229640eca-kserve-provision-location\") pod \"0ec25e63-211c-4d69-865e-8f7229640eca\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") "
Oct 30 22:44:09.093048 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.093037    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec25e63-211c-4d69-865e-8f7229640eca-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ec25e63-211c-4d69-865e-8f7229640eca-cabundle-cert\") pod \"0ec25e63-211c-4d69-865e-8f7229640eca\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") "
Oct 30 22:44:09.093114 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.093066    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ec25e63-211c-4d69-865e-8f7229640eca-proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ec25e63-211c-4d69-865e-8f7229640eca-proxy-tls\") pod \"0ec25e63-211c-4d69-865e-8f7229640eca\" (UID: \"0ec25e63-211c-4d69-865e-8f7229640eca\") "
Oct 30 22:44:09.093316 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.093296    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b049552e-94fe-43a0-b308-e6fc75565d50-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:09.093374 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.093322    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b049552e-94fe-43a0-b308-e6fc75565d50-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:09.093374 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.093339    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b049552e-94fe-43a0-b308-e6fc75565d50-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:09.093374 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.093342    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec25e63-211c-4d69-865e-8f7229640eca-kserve-provision-location" pod "0ec25e63-211c-4d69-865e-8f7229640eca" (UID: "0ec25e63-211c-4d69-865e-8f7229640eca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:09.093374 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.093354    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nfsg\" (UniqueName: \"kubernetes.io/projected/b049552e-94fe-43a0-b308-e6fc75565d50-kube-api-access-7nfsg\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:09.093741 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.093377    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec25e63-211c-4d69-865e-8f7229640eca-kube-rbac-proxy-sar-config" pod "0ec25e63-211c-4d69-865e-8f7229640eca" (UID: "0ec25e63-211c-4d69-865e-8f7229640eca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:09.093741 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.093387    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec25e63-211c-4d69-865e-8f7229640eca-cabundle-cert" pod "0ec25e63-211c-4d69-865e-8f7229640eca" (UID: "0ec25e63-211c-4d69-865e-8f7229640eca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:09.095137 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.095112    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec25e63-211c-4d69-865e-8f7229640eca-kube-api-access-nktfz" pod "0ec25e63-211c-4d69-865e-8f7229640eca" (UID: "0ec25e63-211c-4d69-865e-8f7229640eca"). InnerVolumeSpecName "kube-api-access-nktfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:09.095266 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.095163    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ec25e63-211c-4d69-865e-8f7229640eca-proxy-tls" pod "0ec25e63-211c-4d69-865e-8f7229640eca" (UID: "0ec25e63-211c-4d69-865e-8f7229640eca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:44:09.193875 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.193837    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ec25e63-211c-4d69-865e-8f7229640eca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:09.193875 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.193865    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ec25e63-211c-4d69-865e-8f7229640eca-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:09.193875 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.193876    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ec25e63-211c-4d69-865e-8f7229640eca-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:09.193875 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.193886    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ec25e63-211c-4d69-865e-8f7229640eca-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:09.194222 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.193896    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nktfz\" (UniqueName: \"kubernetes.io/projected/0ec25e63-211c-4d69-865e-8f7229640eca-kube-api-access-nktfz\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:09.956628 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.956550    4473 generic.go:356] "Generic (PLEG): container finished" podID="0ec25e63-211c-4d69-865e-8f7229640eca" containerID="52f8619552c13b7cdd1a5dd6a51162f7f90050113ccfe128ddb322e8416f308e" exitCode=0
Oct 30 22:44:09.956628 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.956609    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"
Oct 30 22:44:09.957179 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.956633    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" event={"ID":"0ec25e63-211c-4d69-865e-8f7229640eca","Type":"ContainerDied","Data":"52f8619552c13b7cdd1a5dd6a51162f7f90050113ccfe128ddb322e8416f308e"}
Oct 30 22:44:09.957179 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.956706    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69" event={"ID":"0ec25e63-211c-4d69-865e-8f7229640eca","Type":"ContainerDied","Data":"812740e71152bcd96ec65434926ec652fea84b0e1eeb6b72e5a684cb040b7a4d"}
Oct 30 22:44:09.957179 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.956747    4473 scope.go:122] "RemoveContainer" containerID="ed003e209e83e8d40535f529ea2eb14e51ab8046b1b3b2630a5aa3e82656aa95"
Oct 30 22:44:09.958652 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.958628    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw" event={"ID":"b049552e-94fe-43a0-b308-e6fc75565d50","Type":"ContainerDied","Data":"a3273d23953efca187d81fd1ec1a394869f53bac28bcb28ea5f4f9aceb59b326"}
Oct 30 22:44:09.958805 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.958697    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"
Oct 30 22:44:09.965300 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.965282    4473 scope.go:122] "RemoveContainer" containerID="52f8619552c13b7cdd1a5dd6a51162f7f90050113ccfe128ddb322e8416f308e"
Oct 30 22:44:09.973009 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.972991    4473 scope.go:122] "RemoveContainer" containerID="6dc08bf8804f5bd45db84087771d227d48f4966f2f976c465cf0fcf5e480960f"
Oct 30 22:44:09.980571 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.980555    4473 scope.go:122] "RemoveContainer" containerID="ed003e209e83e8d40535f529ea2eb14e51ab8046b1b3b2630a5aa3e82656aa95"
Oct 30 22:44:09.980874 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:44:09.980847    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed003e209e83e8d40535f529ea2eb14e51ab8046b1b3b2630a5aa3e82656aa95\": container with ID starting with ed003e209e83e8d40535f529ea2eb14e51ab8046b1b3b2630a5aa3e82656aa95 not found: ID does not exist" containerID="ed003e209e83e8d40535f529ea2eb14e51ab8046b1b3b2630a5aa3e82656aa95"
Oct 30 22:44:09.980968 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.980889    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed003e209e83e8d40535f529ea2eb14e51ab8046b1b3b2630a5aa3e82656aa95"} err="failed to get container status \"ed003e209e83e8d40535f529ea2eb14e51ab8046b1b3b2630a5aa3e82656aa95\": rpc error: code = NotFound desc = could not find container \"ed003e209e83e8d40535f529ea2eb14e51ab8046b1b3b2630a5aa3e82656aa95\": container with ID starting with ed003e209e83e8d40535f529ea2eb14e51ab8046b1b3b2630a5aa3e82656aa95 not found: ID does not exist"
Oct 30 22:44:09.980968 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.980916    4473 scope.go:122] "RemoveContainer" containerID="52f8619552c13b7cdd1a5dd6a51162f7f90050113ccfe128ddb322e8416f308e"
Oct 30 22:44:09.981216 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:44:09.981200    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f8619552c13b7cdd1a5dd6a51162f7f90050113ccfe128ddb322e8416f308e\": container with ID starting with 52f8619552c13b7cdd1a5dd6a51162f7f90050113ccfe128ddb322e8416f308e not found: ID does not exist" containerID="52f8619552c13b7cdd1a5dd6a51162f7f90050113ccfe128ddb322e8416f308e"
Oct 30 22:44:09.981275 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.981223    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f8619552c13b7cdd1a5dd6a51162f7f90050113ccfe128ddb322e8416f308e"} err="failed to get container status \"52f8619552c13b7cdd1a5dd6a51162f7f90050113ccfe128ddb322e8416f308e\": rpc error: code = NotFound desc = could not find container \"52f8619552c13b7cdd1a5dd6a51162f7f90050113ccfe128ddb322e8416f308e\": container with ID starting with 52f8619552c13b7cdd1a5dd6a51162f7f90050113ccfe128ddb322e8416f308e not found: ID does not exist"
Oct 30 22:44:09.981275 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.981237    4473 scope.go:122] "RemoveContainer" containerID="6dc08bf8804f5bd45db84087771d227d48f4966f2f976c465cf0fcf5e480960f"
Oct 30 22:44:09.981469 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:44:09.981452    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc08bf8804f5bd45db84087771d227d48f4966f2f976c465cf0fcf5e480960f\": container with ID starting with 6dc08bf8804f5bd45db84087771d227d48f4966f2f976c465cf0fcf5e480960f not found: ID does not exist" containerID="6dc08bf8804f5bd45db84087771d227d48f4966f2f976c465cf0fcf5e480960f"
Oct 30 22:44:09.981507 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.981480    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc08bf8804f5bd45db84087771d227d48f4966f2f976c465cf0fcf5e480960f"} err="failed to get container status \"6dc08bf8804f5bd45db84087771d227d48f4966f2f976c465cf0fcf5e480960f\": rpc error: code = NotFound desc = could not find container \"6dc08bf8804f5bd45db84087771d227d48f4966f2f976c465cf0fcf5e480960f\": container with ID starting with 6dc08bf8804f5bd45db84087771d227d48f4966f2f976c465cf0fcf5e480960f not found: ID does not exist"
Oct 30 22:44:09.981507 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.981499    4473 scope.go:122] "RemoveContainer" containerID="a7d495c0072eac4c5b47ac1766e68951ffd14a0c56cb3afc0685de5b94924e5f"
Oct 30 22:44:09.989183 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.989092    4473 scope.go:122] "RemoveContainer" containerID="5173af0de539633995e06c8df3bd29e9c101db2ae379c821ddbb82c56b187231"
Oct 30 22:44:09.996639 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:09.996622    4473 scope.go:122] "RemoveContainer" containerID="b05abaa7d786d3734589f6beed2d7d1119934fcc207f114f4970d393c2b695b0"
Oct 30 22:44:10.125697 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:10.125665    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"]
Oct 30 22:44:10.129414 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:10.129393    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-predictor-6d8958c898-7x2fw"]
Oct 30 22:44:10.240538 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:10.240459    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"]
Oct 30 22:44:10.244191 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:10.244163    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-pxs69"]
Oct 30 22:44:11.303966 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:44:11.303911    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:44:11.886906 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:11.886874    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" path="/var/lib/kubelet/pods/0ec25e63-211c-4d69-865e-8f7229640eca/volumes"
Oct 30 22:44:11.887391 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:11.887377    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" path="/var/lib/kubelet/pods/b049552e-94fe-43a0-b308-e6fc75565d50/volumes"
Oct 30 22:44:12.822749 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:12.822709    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:13.022822 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.022783    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:13.172848 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.172817    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:13.438655 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.438572    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"]
Oct 30 22:44:13.438923 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.438910    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kube-rbac-proxy"
Oct 30 22:44:13.439008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.438924    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kube-rbac-proxy"
Oct 30 22:44:13.439008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.438934    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="storage-initializer"
Oct 30 22:44:13.439008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.438939    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="storage-initializer"
Oct 30 22:44:13.439008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.438946    4473 cpu_manager.go:414] "containerMap: removing container" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kube-rbac-proxy"
Oct 30 22:44:13.439008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.438965    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kube-rbac-proxy"
Oct 30 22:44:13.439008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.438975    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kserve-container"
Oct 30 22:44:13.439008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.438979    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kserve-container"
Oct 30 22:44:13.439008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.438986    4473 cpu_manager.go:414] "containerMap: removing container" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kserve-container"
Oct 30 22:44:13.439008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.438991    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kserve-container"
Oct 30 22:44:13.439008 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.439008    4473 cpu_manager.go:414] "containerMap: removing container" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="storage-initializer"
Oct 30 22:44:13.439337 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.439015    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="storage-initializer"
Oct 30 22:44:13.439337 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.439074    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kserve-container"
Oct 30 22:44:13.439337 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.439082    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kserve-container"
Oct 30 22:44:13.439337 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.439093    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b049552e-94fe-43a0-b308-e6fc75565d50" containerName="kube-rbac-proxy"
Oct 30 22:44:13.439337 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.439099    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec25e63-211c-4d69-865e-8f7229640eca" containerName="kube-rbac-proxy"
Oct 30 22:44:13.443966 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.443939    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:13.498111 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.498081    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Oct 30 22:44:13.530505 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.530470    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d521379-4169-4a34-89c0-2927c189a262-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:13.530630 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.530508    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsf25\" (UniqueName: \"kubernetes.io/projected/2d521379-4169-4a34-89c0-2927c189a262-kube-api-access-lsf25\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:13.530630 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.530536    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d521379-4169-4a34-89c0-2927c189a262-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:13.530630 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.530619    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d521379-4169-4a34-89c0-2927c189a262-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:13.530741 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.530686    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d521379-4169-4a34-89c0-2927c189a262-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:13.548229 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.548188    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"storage-config\""
Oct 30 22:44:13.548434 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.548413    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:44:13.548531 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.548413    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:44:13.548531 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.548450    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Oct 30 22:44:13.548531 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.548456    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:44:13.548625 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.548540    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:44:13.549917 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.549899    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"]
Oct 30 22:44:13.631901 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.631864    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d521379-4169-4a34-89c0-2927c189a262-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:13.632147 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.631914    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d521379-4169-4a34-89c0-2927c189a262-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:13.632147 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.632066    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsf25\" (UniqueName: \"kubernetes.io/projected/2d521379-4169-4a34-89c0-2927c189a262-kube-api-access-lsf25\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:13.632147 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.632121    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d521379-4169-4a34-89c0-2927c189a262-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:13.632324 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.632151    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d521379-4169-4a34-89c0-2927c189a262-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:13.632569 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.632545    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d521379-4169-4a34-89c0-2927c189a262-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:13.632639 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.632595    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d521379-4169-4a34-89c0-2927c189a262-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:13.632735 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.632716    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d521379-4169-4a34-89c0-2927c189a262-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:13.634826 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.634806    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d521379-4169-4a34-89c0-2927c189a262-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:13.691261 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.691192    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsf25\" (UniqueName: \"kubernetes.io/projected/2d521379-4169-4a34-89c0-2927c189a262-kube-api-access-lsf25\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:13.761540 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.761504    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:13.982548 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.982471    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"]
Oct 30 22:44:13.982974 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.982713    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kserve-container" containerID="cri-o://cd01803dbf487ce362aeaf14879e56edd402fa7da62ce5cb80fc9c17d461863d" gracePeriod=30
Oct 30 22:44:13.982974 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:13.982783    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kube-rbac-proxy" containerID="cri-o://351d7fb62ac249658bebd3c6cb207fe8be7c3b967c52d341576e05f211b79796" gracePeriod=30
Oct 30 22:44:14.154755 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.154716    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:14.201394 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.201369    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"]
Oct 30 22:44:14.203595 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:44:14.203568    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d521379_4169_4a34_89c0_2927c189a262.slice/crio-ee4dfab04ca79ed29d6c0ba5cfb6ed496fdeb33f925712692e1d6cdd70a0e2cd WatchSource:0}: Error finding container ee4dfab04ca79ed29d6c0ba5cfb6ed496fdeb33f925712692e1d6cdd70a0e2cd: Status 404 returned error can't find the container with id ee4dfab04ca79ed29d6c0ba5cfb6ed496fdeb33f925712692e1d6cdd70a0e2cd
Oct 30 22:44:14.323612 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.323573    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"]
Oct 30 22:44:14.327082 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.327067    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:14.336647 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.336620    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9228c263-f2fa-4e48-9dc4-52face15d99f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-wfpjg\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:14.336762 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.336668    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49gg5\" (UniqueName: \"kubernetes.io/projected/9228c263-f2fa-4e48-9dc4-52face15d99f-kube-api-access-49gg5\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-wfpjg\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:14.336819 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.336756    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9228c263-f2fa-4e48-9dc4-52face15d99f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-wfpjg\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:14.336879 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.336862    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9228c263-f2fa-4e48-9dc4-52face15d99f-proxy-tls\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-wfpjg\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:14.336924 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.336894    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9228c263-f2fa-4e48-9dc4-52face15d99f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-wfpjg\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:14.380558 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.380530    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-canary-rollback-predictor-serving-cert\""
Oct 30 22:44:14.388099 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.388074    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"]
Oct 30 22:44:14.437375 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.437346    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9228c263-f2fa-4e48-9dc4-52face15d99f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-wfpjg\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:14.437375 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.437382    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49gg5\" (UniqueName: \"kubernetes.io/projected/9228c263-f2fa-4e48-9dc4-52face15d99f-kube-api-access-49gg5\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-wfpjg\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:14.437633 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.437404    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9228c263-f2fa-4e48-9dc4-52face15d99f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-wfpjg\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:14.437633 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.437460    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9228c263-f2fa-4e48-9dc4-52face15d99f-proxy-tls\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-wfpjg\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:14.437633 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.437482    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9228c263-f2fa-4e48-9dc4-52face15d99f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-wfpjg\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:14.438069 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.438044    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9228c263-f2fa-4e48-9dc4-52face15d99f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-wfpjg\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:14.438184 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.438159    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9228c263-f2fa-4e48-9dc4-52face15d99f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-wfpjg\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:14.438238 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.438191    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9228c263-f2fa-4e48-9dc4-52face15d99f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-wfpjg\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:14.440800 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.440778    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9228c263-f2fa-4e48-9dc4-52face15d99f-proxy-tls\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-wfpjg\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:14.496097 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.496024    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49gg5\" (UniqueName: \"kubernetes.io/projected/9228c263-f2fa-4e48-9dc4-52face15d99f-kube-api-access-49gg5\") pod \"isvc-canary-rollback-predictor-78cd59d4dc-wfpjg\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") " pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:14.644279 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.644240    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:14.983550 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.983509    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" event={"ID":"2d521379-4169-4a34-89c0-2927c189a262","Type":"ContainerStarted","Data":"2fa6a794a68bd5f2264edff99258019c27af48c97ea961e1b8fcaba9a8c63b6e"}
Oct 30 22:44:14.984033 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.983586    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" event={"ID":"2d521379-4169-4a34-89c0-2927c189a262","Type":"ContainerStarted","Data":"ee4dfab04ca79ed29d6c0ba5cfb6ed496fdeb33f925712692e1d6cdd70a0e2cd"}
Oct 30 22:44:14.985685 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.985661    4473 generic.go:356] "Generic (PLEG): container finished" podID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerID="351d7fb62ac249658bebd3c6cb207fe8be7c3b967c52d341576e05f211b79796" exitCode=0
Oct 30 22:44:14.985803 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:14.985709    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" event={"ID":"5f38a5e6-8045-40c9-b61d-545e80d440e7","Type":"ContainerDied","Data":"351d7fb62ac249658bebd3c6cb207fe8be7c3b967c52d341576e05f211b79796"}
Oct 30 22:44:15.073644 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:15.073620    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"]
Oct 30 22:44:15.075615 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:44:15.075585    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9228c263_f2fa_4e48_9dc4_52face15d99f.slice/crio-3a369cf81d80f8200801ab708ee39cdd6ff2db910637a691f9b36d4c32ad95ce WatchSource:0}: Error finding container 3a369cf81d80f8200801ab708ee39cdd6ff2db910637a691f9b36d4c32ad95ce: Status 404 returned error can't find the container with id 3a369cf81d80f8200801ab708ee39cdd6ff2db910637a691f9b36d4c32ad95ce
Oct 30 22:44:15.991817 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:15.991769    4473 generic.go:356] "Generic (PLEG): container finished" podID="2d521379-4169-4a34-89c0-2927c189a262" containerID="2fa6a794a68bd5f2264edff99258019c27af48c97ea961e1b8fcaba9a8c63b6e" exitCode=0
Oct 30 22:44:15.992286 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:15.991852    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" event={"ID":"2d521379-4169-4a34-89c0-2927c189a262","Type":"ContainerDied","Data":"2fa6a794a68bd5f2264edff99258019c27af48c97ea961e1b8fcaba9a8c63b6e"}
Oct 30 22:44:15.993433 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:15.993409    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" event={"ID":"9228c263-f2fa-4e48-9dc4-52face15d99f","Type":"ContainerStarted","Data":"3c234e5078d7da762893aa331fcfcc539e91e25f7565115156c24c1126fedd70"}
Oct 30 22:44:15.993551 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:15.993462    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" event={"ID":"9228c263-f2fa-4e48-9dc4-52face15d99f","Type":"ContainerStarted","Data":"3a369cf81d80f8200801ab708ee39cdd6ff2db910637a691f9b36d4c32ad95ce"}
Oct 30 22:44:16.563478 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:16.563376    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Oct 30 22:44:17.006131 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:17.006095    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" event={"ID":"2d521379-4169-4a34-89c0-2927c189a262","Type":"ContainerStarted","Data":"a8d667b2d0170fe1fb5653320eb3f37cf026534f1e6d142ca11ae5ef7d7d0fe8"}
Oct 30 22:44:17.006612 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:17.006292    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:17.006612 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:17.006347    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" event={"ID":"2d521379-4169-4a34-89c0-2927c189a262","Type":"ContainerStarted","Data":"aeeef39fdcb946d2305652497a74224a404d88af5778ba0193aee64bb20b6d51"}
Oct 30 22:44:17.006612 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:17.006386    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:17.007584 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:17.007556    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Oct 30 22:44:17.127734 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:17.127674    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" podStartSLOduration=3.052191022 podStartE2EDuration="4.127657271s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.075466249s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:17.127179797 +0000 UTC m=+3075.363389143" watchObservedRunningTime="2026-10-30 22:44:17.127657271 +0000 UTC m=+3075.363866618"
Oct 30 22:44:18.010393 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:18.010355    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Oct 30 22:44:18.943268 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:18.943243    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:44:18.981053 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:18.981028    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f38a5e6-8045-40c9-b61d-545e80d440e7-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f38a5e6-8045-40c9-b61d-545e80d440e7-cabundle-cert\") pod \"5f38a5e6-8045-40c9-b61d-545e80d440e7\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") "
Oct 30 22:44:18.981150 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:18.981094    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f38a5e6-8045-40c9-b61d-545e80d440e7-proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f38a5e6-8045-40c9-b61d-545e80d440e7-proxy-tls\") pod \"5f38a5e6-8045-40c9-b61d-545e80d440e7\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") "
Oct 30 22:44:18.981150 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:18.981115    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f38a5e6-8045-40c9-b61d-545e80d440e7-kube-api-access-g697c\" (UniqueName: \"kubernetes.io/projected/5f38a5e6-8045-40c9-b61d-545e80d440e7-kube-api-access-g697c\") pod \"5f38a5e6-8045-40c9-b61d-545e80d440e7\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") "
Oct 30 22:44:18.981244 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:18.981160    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f38a5e6-8045-40c9-b61d-545e80d440e7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f38a5e6-8045-40c9-b61d-545e80d440e7-kube-rbac-proxy-sar-config\") pod \"5f38a5e6-8045-40c9-b61d-545e80d440e7\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") "
Oct 30 22:44:18.981244 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:18.981191    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f38a5e6-8045-40c9-b61d-545e80d440e7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f38a5e6-8045-40c9-b61d-545e80d440e7-kserve-provision-location\") pod \"5f38a5e6-8045-40c9-b61d-545e80d440e7\" (UID: \"5f38a5e6-8045-40c9-b61d-545e80d440e7\") "
Oct 30 22:44:18.981509 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:18.981477    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f38a5e6-8045-40c9-b61d-545e80d440e7-cabundle-cert" pod "5f38a5e6-8045-40c9-b61d-545e80d440e7" (UID: "5f38a5e6-8045-40c9-b61d-545e80d440e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:18.981632 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:18.981523    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f38a5e6-8045-40c9-b61d-545e80d440e7-kube-rbac-proxy-sar-config" pod "5f38a5e6-8045-40c9-b61d-545e80d440e7" (UID: "5f38a5e6-8045-40c9-b61d-545e80d440e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:18.981632 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:18.981537    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f38a5e6-8045-40c9-b61d-545e80d440e7-kserve-provision-location" pod "5f38a5e6-8045-40c9-b61d-545e80d440e7" (UID: "5f38a5e6-8045-40c9-b61d-545e80d440e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:18.983260 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:18.983240    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f38a5e6-8045-40c9-b61d-545e80d440e7-proxy-tls" pod "5f38a5e6-8045-40c9-b61d-545e80d440e7" (UID: "5f38a5e6-8045-40c9-b61d-545e80d440e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:44:18.983477 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:18.983459    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f38a5e6-8045-40c9-b61d-545e80d440e7-kube-api-access-g697c" pod "5f38a5e6-8045-40c9-b61d-545e80d440e7" (UID: "5f38a5e6-8045-40c9-b61d-545e80d440e7"). InnerVolumeSpecName "kube-api-access-g697c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:19.016166 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.016138    4473 generic.go:356] "Generic (PLEG): container finished" podID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerID="3c234e5078d7da762893aa331fcfcc539e91e25f7565115156c24c1126fedd70" exitCode=0
Oct 30 22:44:19.016580 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.016177    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" event={"ID":"9228c263-f2fa-4e48-9dc4-52face15d99f","Type":"ContainerDied","Data":"3c234e5078d7da762893aa331fcfcc539e91e25f7565115156c24c1126fedd70"}
Oct 30 22:44:19.018204 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.018182    4473 generic.go:356] "Generic (PLEG): container finished" podID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerID="cd01803dbf487ce362aeaf14879e56edd402fa7da62ce5cb80fc9c17d461863d" exitCode=0
Oct 30 22:44:19.018306 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.018244    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"
Oct 30 22:44:19.018306 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.018252    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" event={"ID":"5f38a5e6-8045-40c9-b61d-545e80d440e7","Type":"ContainerDied","Data":"cd01803dbf487ce362aeaf14879e56edd402fa7da62ce5cb80fc9c17d461863d"}
Oct 30 22:44:19.018427 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.018305    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph" event={"ID":"5f38a5e6-8045-40c9-b61d-545e80d440e7","Type":"ContainerDied","Data":"6a6b2ab55f62c065d86c02fd0f052abc134d2d397bb23e2ead5831677f0e28e2"}
Oct 30 22:44:19.018427 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.018339    4473 scope.go:122] "RemoveContainer" containerID="351d7fb62ac249658bebd3c6cb207fe8be7c3b967c52d341576e05f211b79796"
Oct 30 22:44:19.043343 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.043325    4473 scope.go:122] "RemoveContainer" containerID="cd01803dbf487ce362aeaf14879e56edd402fa7da62ce5cb80fc9c17d461863d"
Oct 30 22:44:19.065217 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.065191    4473 scope.go:122] "RemoveContainer" containerID="cfed2820bd0cf34e31d09c24b851b45f15fcd4cc6df27591186e8588f87b8167"
Oct 30 22:44:19.073760 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.073741    4473 scope.go:122] "RemoveContainer" containerID="351d7fb62ac249658bebd3c6cb207fe8be7c3b967c52d341576e05f211b79796"
Oct 30 22:44:19.074075 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:44:19.074052    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"351d7fb62ac249658bebd3c6cb207fe8be7c3b967c52d341576e05f211b79796\": container with ID starting with 351d7fb62ac249658bebd3c6cb207fe8be7c3b967c52d341576e05f211b79796 not found: ID does not exist" containerID="351d7fb62ac249658bebd3c6cb207fe8be7c3b967c52d341576e05f211b79796"
Oct 30 22:44:19.074174 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.074082    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"351d7fb62ac249658bebd3c6cb207fe8be7c3b967c52d341576e05f211b79796"} err="failed to get container status \"351d7fb62ac249658bebd3c6cb207fe8be7c3b967c52d341576e05f211b79796\": rpc error: code = NotFound desc = could not find container \"351d7fb62ac249658bebd3c6cb207fe8be7c3b967c52d341576e05f211b79796\": container with ID starting with 351d7fb62ac249658bebd3c6cb207fe8be7c3b967c52d341576e05f211b79796 not found: ID does not exist"
Oct 30 22:44:19.074174 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.074102    4473 scope.go:122] "RemoveContainer" containerID="cd01803dbf487ce362aeaf14879e56edd402fa7da62ce5cb80fc9c17d461863d"
Oct 30 22:44:19.074369 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:44:19.074352    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd01803dbf487ce362aeaf14879e56edd402fa7da62ce5cb80fc9c17d461863d\": container with ID starting with cd01803dbf487ce362aeaf14879e56edd402fa7da62ce5cb80fc9c17d461863d not found: ID does not exist" containerID="cd01803dbf487ce362aeaf14879e56edd402fa7da62ce5cb80fc9c17d461863d"
Oct 30 22:44:19.074472 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.074373    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd01803dbf487ce362aeaf14879e56edd402fa7da62ce5cb80fc9c17d461863d"} err="failed to get container status \"cd01803dbf487ce362aeaf14879e56edd402fa7da62ce5cb80fc9c17d461863d\": rpc error: code = NotFound desc = could not find container \"cd01803dbf487ce362aeaf14879e56edd402fa7da62ce5cb80fc9c17d461863d\": container with ID starting with cd01803dbf487ce362aeaf14879e56edd402fa7da62ce5cb80fc9c17d461863d not found: ID does not exist"
Oct 30 22:44:19.074472 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.074385    4473 scope.go:122] "RemoveContainer" containerID="cfed2820bd0cf34e31d09c24b851b45f15fcd4cc6df27591186e8588f87b8167"
Oct 30 22:44:19.074590 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:44:19.074574    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfed2820bd0cf34e31d09c24b851b45f15fcd4cc6df27591186e8588f87b8167\": container with ID starting with cfed2820bd0cf34e31d09c24b851b45f15fcd4cc6df27591186e8588f87b8167 not found: ID does not exist" containerID="cfed2820bd0cf34e31d09c24b851b45f15fcd4cc6df27591186e8588f87b8167"
Oct 30 22:44:19.074628 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.074595    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfed2820bd0cf34e31d09c24b851b45f15fcd4cc6df27591186e8588f87b8167"} err="failed to get container status \"cfed2820bd0cf34e31d09c24b851b45f15fcd4cc6df27591186e8588f87b8167\": rpc error: code = NotFound desc = could not find container \"cfed2820bd0cf34e31d09c24b851b45f15fcd4cc6df27591186e8588f87b8167\": container with ID starting with cfed2820bd0cf34e31d09c24b851b45f15fcd4cc6df27591186e8588f87b8167 not found: ID does not exist"
Oct 30 22:44:19.082466 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.082445    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f38a5e6-8045-40c9-b61d-545e80d440e7-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:19.082553 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.082473    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g697c\" (UniqueName: \"kubernetes.io/projected/5f38a5e6-8045-40c9-b61d-545e80d440e7-kube-api-access-g697c\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:19.082553 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.082488    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f38a5e6-8045-40c9-b61d-545e80d440e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:19.082553 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.082502    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f38a5e6-8045-40c9-b61d-545e80d440e7-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:19.082553 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.082516    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f38a5e6-8045-40c9-b61d-545e80d440e7-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:19.355429 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.355346    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"]
Oct 30 22:44:19.359198 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.359173    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-v2-predictor-6646b5d9fd-r97ph"]
Oct 30 22:44:19.886788 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:19.886755    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" path="/var/lib/kubelet/pods/5f38a5e6-8045-40c9-b61d-545e80d440e7/volumes"
Oct 30 22:44:20.024672 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:20.024632    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" event={"ID":"9228c263-f2fa-4e48-9dc4-52face15d99f","Type":"ContainerStarted","Data":"1466623da436525b0bbfdcdec08d617a5ff0bf570f62bffcd8b2e8e04f342d8c"}
Oct 30 22:44:20.025121 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:20.024699    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" event={"ID":"9228c263-f2fa-4e48-9dc4-52face15d99f","Type":"ContainerStarted","Data":"7966fa82ee786db69386f1ea472ca14f538fb18d18069ca76fc76206914f6ad5"}
Oct 30 22:44:20.025121 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:20.024995    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:20.025121 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:20.025110    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:20.026644 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:20.026615    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Oct 30 22:44:20.198563 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:20.198510    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" podStartSLOduration=2.36473809 podStartE2EDuration="6.19849512s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.83375703s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:20.198075627 +0000 UTC m=+3078.434284980" watchObservedRunningTime="2026-10-30 22:44:20.19849512 +0000 UTC m=+3078.434704465"
Oct 30 22:44:21.030224 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:21.030186    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Oct 30 22:44:23.015571 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:23.015545    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:44:23.016097 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:23.016073    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Oct 30 22:44:26.035629 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:26.035599    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:44:26.036142 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:26.036118    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Oct 30 22:44:33.016712 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:33.016672    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Oct 30 22:44:36.036681 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:36.036640    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Oct 30 22:44:41.303844 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:44:41.303805    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:44:43.016101 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:43.016014    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Oct 30 22:44:46.036874 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:46.036830    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Oct 30 22:44:53.016915 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:53.016876    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Oct 30 22:44:56.036908 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:44:56.036872    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Oct 30 22:45:03.016653 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:03.016608    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Oct 30 22:45:06.036093 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:06.036047    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Oct 30 22:45:08.853847 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:08.853805    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:11.303650 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:45:11.303610    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:45:13.016619 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:13.016575    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Oct 30 22:45:16.036736 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:16.036694    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Oct 30 22:45:23.016424 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:23.016380    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Oct 30 22:45:26.036329 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:26.036290    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Oct 30 22:45:33.017166 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:33.017128    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:45:36.036164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:36.036121    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Oct 30 22:45:38.882640 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:38.882610    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:45:41.304145 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:45:41.304106    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:45:42.302726 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:42.302691    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"]
Oct 30 22:45:42.302993 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:42.302968    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kserve-container" containerID="cri-o://aeeef39fdcb946d2305652497a74224a404d88af5778ba0193aee64bb20b6d51" gracePeriod=30
Oct 30 22:45:42.303078 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:42.303034    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kube-rbac-proxy" containerID="cri-o://a8d667b2d0170fe1fb5653320eb3f37cf026534f1e6d142ca11ae5ef7d7d0fe8" gracePeriod=30
Oct 30 22:45:42.311535 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:42.311495    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:43.010896 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:43.010852    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Oct 30 22:45:43.016151 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:43.016127    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Oct 30 22:45:43.387342 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:43.387243    4473 generic.go:356] "Generic (PLEG): container finished" podID="2d521379-4169-4a34-89c0-2927c189a262" containerID="a8d667b2d0170fe1fb5653320eb3f37cf026534f1e6d142ca11ae5ef7d7d0fe8" exitCode=0
Oct 30 22:45:43.387342 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:43.387316    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" event={"ID":"2d521379-4169-4a34-89c0-2927c189a262","Type":"ContainerDied","Data":"a8d667b2d0170fe1fb5653320eb3f37cf026534f1e6d142ca11ae5ef7d7d0fe8"}
Oct 30 22:45:43.570685 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:43.570656    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"]
Oct 30 22:45:43.570944 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:43.570910    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kserve-container" containerID="cri-o://7966fa82ee786db69386f1ea472ca14f538fb18d18069ca76fc76206914f6ad5" gracePeriod=30
Oct 30 22:45:43.571065 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:43.570937    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kube-rbac-proxy" containerID="cri-o://1466623da436525b0bbfdcdec08d617a5ff0bf570f62bffcd8b2e8e04f342d8c" gracePeriod=30
Oct 30 22:45:43.592755 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:43.592732    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:43.746770 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:43.746729    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:44.343086 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:44.343050    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:44.394131 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:44.394095    4473 generic.go:356] "Generic (PLEG): container finished" podID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerID="1466623da436525b0bbfdcdec08d617a5ff0bf570f62bffcd8b2e8e04f342d8c" exitCode=0
Oct 30 22:45:44.394507 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:44.394143    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" event={"ID":"9228c263-f2fa-4e48-9dc4-52face15d99f","Type":"ContainerDied","Data":"1466623da436525b0bbfdcdec08d617a5ff0bf570f62bffcd8b2e8e04f342d8c"}
Oct 30 22:45:44.541615 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:44.541577    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:44.955163 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:44.955122    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"]
Oct 30 22:45:44.955495 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:44.955484    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kserve-container"
Oct 30 22:45:44.955539 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:44.955497    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kserve-container"
Oct 30 22:45:44.955539 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:44.955512    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="storage-initializer"
Oct 30 22:45:44.955539 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:44.955518    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="storage-initializer"
Oct 30 22:45:44.955539 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:44.955530    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kube-rbac-proxy"
Oct 30 22:45:44.955539 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:44.955535    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kube-rbac-proxy"
Oct 30 22:45:44.955687 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:44.955600    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kserve-container"
Oct 30 22:45:44.955687 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:44.955609    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f38a5e6-8045-40c9-b61d-545e80d440e7" containerName="kube-rbac-proxy"
Oct 30 22:45:44.958562 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:44.958547    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"
Oct 30 22:45:45.011839 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:45.011811    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Oct 30 22:45:45.062313 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:45.062278    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Oct 30 22:45:45.064310 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:45.064276    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"]
Oct 30 22:45:45.114762 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:45.114722    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8\" (UID: \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"
Oct 30 22:45:45.114936 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:45.114836    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8\" (UID: \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"
Oct 30 22:45:45.114936 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:45.114879    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8\" (UID: \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"
Oct 30 22:45:45.114936 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:45.114900    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmng7\" (UniqueName: \"kubernetes.io/projected/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kube-api-access-pmng7\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8\" (UID: \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"
Oct 30 22:45:45.215348 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:45.215261    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8\" (UID: \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"
Oct 30 22:45:45.215348 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:45.215316    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8\" (UID: \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"
Oct 30 22:45:45.215560 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:45.215350    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8\" (UID: \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"
Oct 30 22:45:45.215560 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:45.215369    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmng7\" (UniqueName: \"kubernetes.io/projected/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kube-api-access-pmng7\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8\" (UID: \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"
Oct 30 22:45:45.215745 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:45.215718    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8\" (UID: \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"
Oct 30 22:45:45.216030 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:45.216008    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8\" (UID: \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"
Oct 30 22:45:45.217926 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:45.217911    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8\" (UID: \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"
Oct 30 22:45:45.273582 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:45.273545    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmng7\" (UniqueName: \"kubernetes.io/projected/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kube-api-access-pmng7\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8\" (UID: \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"
Oct 30 22:45:45.277392 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:45.277364    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"
Oct 30 22:45:45.704246 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:45.704222    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"]
Oct 30 22:45:45.706684 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:45:45.706655    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f1b7d9_c087_432e_bed8_5c0cf63d0eda.slice/crio-126555ae8097510052bfd42942604dcdfe1055049368f75872021a805ba0cd56 WatchSource:0}: Error finding container 126555ae8097510052bfd42942604dcdfe1055049368f75872021a805ba0cd56: Status 404 returned error can't find the container with id 126555ae8097510052bfd42942604dcdfe1055049368f75872021a805ba0cd56
Oct 30 22:45:46.031369 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:46.031277    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
Oct 30 22:45:46.403749 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:46.403713    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8" event={"ID":"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda","Type":"ContainerStarted","Data":"e07cfdd09411b1070f31f883a4dd52aa0db8b7e089692db3dab6448ebb9c6fc0"}
Oct 30 22:45:46.403977 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:46.403767    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8" event={"ID":"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda","Type":"ContainerStarted","Data":"126555ae8097510052bfd42942604dcdfe1055049368f75872021a805ba0cd56"}
Oct 30 22:45:47.561611 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:47.561585    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:45:47.737861 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:47.737831    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d521379-4169-4a34-89c0-2927c189a262-proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d521379-4169-4a34-89c0-2927c189a262-proxy-tls\") pod \"2d521379-4169-4a34-89c0-2927c189a262\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") "
Oct 30 22:45:47.738065 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:47.737878    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d521379-4169-4a34-89c0-2927c189a262-kube-api-access-lsf25\" (UniqueName: \"kubernetes.io/projected/2d521379-4169-4a34-89c0-2927c189a262-kube-api-access-lsf25\") pod \"2d521379-4169-4a34-89c0-2927c189a262\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") "
Oct 30 22:45:47.738065 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:47.737908    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d521379-4169-4a34-89c0-2927c189a262-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d521379-4169-4a34-89c0-2927c189a262-kube-rbac-proxy-sar-config\") pod \"2d521379-4169-4a34-89c0-2927c189a262\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") "
Oct 30 22:45:47.738065 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:47.737925    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d521379-4169-4a34-89c0-2927c189a262-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d521379-4169-4a34-89c0-2927c189a262-kserve-provision-location\") pod \"2d521379-4169-4a34-89c0-2927c189a262\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") "
Oct 30 22:45:47.738065 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:47.737946    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d521379-4169-4a34-89c0-2927c189a262-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d521379-4169-4a34-89c0-2927c189a262-cabundle-cert\") pod \"2d521379-4169-4a34-89c0-2927c189a262\" (UID: \"2d521379-4169-4a34-89c0-2927c189a262\") "
Oct 30 22:45:47.738316 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:47.738281    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d521379-4169-4a34-89c0-2927c189a262-kserve-provision-location" pod "2d521379-4169-4a34-89c0-2927c189a262" (UID: "2d521379-4169-4a34-89c0-2927c189a262"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:47.738443 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:47.738315    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d521379-4169-4a34-89c0-2927c189a262-kube-rbac-proxy-sar-config" pod "2d521379-4169-4a34-89c0-2927c189a262" (UID: "2d521379-4169-4a34-89c0-2927c189a262"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:47.738443 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:47.738390    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d521379-4169-4a34-89c0-2927c189a262-cabundle-cert" pod "2d521379-4169-4a34-89c0-2927c189a262" (UID: "2d521379-4169-4a34-89c0-2927c189a262"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:47.740244 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:47.740223    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d521379-4169-4a34-89c0-2927c189a262-proxy-tls" pod "2d521379-4169-4a34-89c0-2927c189a262" (UID: "2d521379-4169-4a34-89c0-2927c189a262"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:47.740340 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:47.740327    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d521379-4169-4a34-89c0-2927c189a262-kube-api-access-lsf25" pod "2d521379-4169-4a34-89c0-2927c189a262" (UID: "2d521379-4169-4a34-89c0-2927c189a262"). InnerVolumeSpecName "kube-api-access-lsf25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:47.839392 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:47.839335    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d521379-4169-4a34-89c0-2927c189a262-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:47.839392 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:47.839385    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d521379-4169-4a34-89c0-2927c189a262-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:47.839392 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:47.839399    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d521379-4169-4a34-89c0-2927c189a262-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:47.839638 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:47.839413    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d521379-4169-4a34-89c0-2927c189a262-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:47.839638 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:47.839425    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsf25\" (UniqueName: \"kubernetes.io/projected/2d521379-4169-4a34-89c0-2927c189a262-kube-api-access-lsf25\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:48.414056 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.414025    4473 generic.go:356] "Generic (PLEG): container finished" podID="2d521379-4169-4a34-89c0-2927c189a262" containerID="aeeef39fdcb946d2305652497a74224a404d88af5778ba0193aee64bb20b6d51" exitCode=0
Oct 30 22:45:48.414233 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.414070    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" event={"ID":"2d521379-4169-4a34-89c0-2927c189a262","Type":"ContainerDied","Data":"aeeef39fdcb946d2305652497a74224a404d88af5778ba0193aee64bb20b6d51"}
Oct 30 22:45:48.414233 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.414087    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"
Oct 30 22:45:48.414233 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.414130    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc" event={"ID":"2d521379-4169-4a34-89c0-2927c189a262","Type":"ContainerDied","Data":"ee4dfab04ca79ed29d6c0ba5cfb6ed496fdeb33f925712692e1d6cdd70a0e2cd"}
Oct 30 22:45:48.414233 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.414160    4473 scope.go:122] "RemoveContainer" containerID="a8d667b2d0170fe1fb5653320eb3f37cf026534f1e6d142ca11ae5ef7d7d0fe8"
Oct 30 22:45:48.422705 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.422555    4473 scope.go:122] "RemoveContainer" containerID="aeeef39fdcb946d2305652497a74224a404d88af5778ba0193aee64bb20b6d51"
Oct 30 22:45:48.429885 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.429867    4473 scope.go:122] "RemoveContainer" containerID="2fa6a794a68bd5f2264edff99258019c27af48c97ea961e1b8fcaba9a8c63b6e"
Oct 30 22:45:48.437207 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.437186    4473 scope.go:122] "RemoveContainer" containerID="a8d667b2d0170fe1fb5653320eb3f37cf026534f1e6d142ca11ae5ef7d7d0fe8"
Oct 30 22:45:48.437475 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:45:48.437451    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d667b2d0170fe1fb5653320eb3f37cf026534f1e6d142ca11ae5ef7d7d0fe8\": container with ID starting with a8d667b2d0170fe1fb5653320eb3f37cf026534f1e6d142ca11ae5ef7d7d0fe8 not found: ID does not exist" containerID="a8d667b2d0170fe1fb5653320eb3f37cf026534f1e6d142ca11ae5ef7d7d0fe8"
Oct 30 22:45:48.437513 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.437487    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d667b2d0170fe1fb5653320eb3f37cf026534f1e6d142ca11ae5ef7d7d0fe8"} err="failed to get container status \"a8d667b2d0170fe1fb5653320eb3f37cf026534f1e6d142ca11ae5ef7d7d0fe8\": rpc error: code = NotFound desc = could not find container \"a8d667b2d0170fe1fb5653320eb3f37cf026534f1e6d142ca11ae5ef7d7d0fe8\": container with ID starting with a8d667b2d0170fe1fb5653320eb3f37cf026534f1e6d142ca11ae5ef7d7d0fe8 not found: ID does not exist"
Oct 30 22:45:48.437551 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.437518    4473 scope.go:122] "RemoveContainer" containerID="aeeef39fdcb946d2305652497a74224a404d88af5778ba0193aee64bb20b6d51"
Oct 30 22:45:48.437771 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:45:48.437755    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeeef39fdcb946d2305652497a74224a404d88af5778ba0193aee64bb20b6d51\": container with ID starting with aeeef39fdcb946d2305652497a74224a404d88af5778ba0193aee64bb20b6d51 not found: ID does not exist" containerID="aeeef39fdcb946d2305652497a74224a404d88af5778ba0193aee64bb20b6d51"
Oct 30 22:45:48.437811 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.437777    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeeef39fdcb946d2305652497a74224a404d88af5778ba0193aee64bb20b6d51"} err="failed to get container status \"aeeef39fdcb946d2305652497a74224a404d88af5778ba0193aee64bb20b6d51\": rpc error: code = NotFound desc = could not find container \"aeeef39fdcb946d2305652497a74224a404d88af5778ba0193aee64bb20b6d51\": container with ID starting with aeeef39fdcb946d2305652497a74224a404d88af5778ba0193aee64bb20b6d51 not found: ID does not exist"
Oct 30 22:45:48.437811 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.437792    4473 scope.go:122] "RemoveContainer" containerID="2fa6a794a68bd5f2264edff99258019c27af48c97ea961e1b8fcaba9a8c63b6e"
Oct 30 22:45:48.438038 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:45:48.438023    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa6a794a68bd5f2264edff99258019c27af48c97ea961e1b8fcaba9a8c63b6e\": container with ID starting with 2fa6a794a68bd5f2264edff99258019c27af48c97ea961e1b8fcaba9a8c63b6e not found: ID does not exist" containerID="2fa6a794a68bd5f2264edff99258019c27af48c97ea961e1b8fcaba9a8c63b6e"
Oct 30 22:45:48.438088 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.438043    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa6a794a68bd5f2264edff99258019c27af48c97ea961e1b8fcaba9a8c63b6e"} err="failed to get container status \"2fa6a794a68bd5f2264edff99258019c27af48c97ea961e1b8fcaba9a8c63b6e\": rpc error: code = NotFound desc = could not find container \"2fa6a794a68bd5f2264edff99258019c27af48c97ea961e1b8fcaba9a8c63b6e\": container with ID starting with 2fa6a794a68bd5f2264edff99258019c27af48c97ea961e1b8fcaba9a8c63b6e not found: ID does not exist"
Oct 30 22:45:48.631558 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.631525    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"]
Oct 30 22:45:48.633794 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.633765    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-pass-predictor-68bc985cdd-f8ljc"]
Oct 30 22:45:48.712295 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.712268    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:45:48.847380 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.847347    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9228c263-f2fa-4e48-9dc4-52face15d99f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9228c263-f2fa-4e48-9dc4-52face15d99f-kube-rbac-proxy-sar-config\") pod \"9228c263-f2fa-4e48-9dc4-52face15d99f\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") "
Oct 30 22:45:48.847580 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.847436    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9228c263-f2fa-4e48-9dc4-52face15d99f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9228c263-f2fa-4e48-9dc4-52face15d99f-kserve-provision-location\") pod \"9228c263-f2fa-4e48-9dc4-52face15d99f\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") "
Oct 30 22:45:48.847580 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.847475    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9228c263-f2fa-4e48-9dc4-52face15d99f-kube-api-access-49gg5\" (UniqueName: \"kubernetes.io/projected/9228c263-f2fa-4e48-9dc4-52face15d99f-kube-api-access-49gg5\") pod \"9228c263-f2fa-4e48-9dc4-52face15d99f\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") "
Oct 30 22:45:48.847580 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.847538    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9228c263-f2fa-4e48-9dc4-52face15d99f-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9228c263-f2fa-4e48-9dc4-52face15d99f-proxy-tls\") pod \"9228c263-f2fa-4e48-9dc4-52face15d99f\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") "
Oct 30 22:45:48.847580 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.847562    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9228c263-f2fa-4e48-9dc4-52face15d99f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9228c263-f2fa-4e48-9dc4-52face15d99f-cabundle-cert\") pod \"9228c263-f2fa-4e48-9dc4-52face15d99f\" (UID: \"9228c263-f2fa-4e48-9dc4-52face15d99f\") "
Oct 30 22:45:48.847788 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.847767    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9228c263-f2fa-4e48-9dc4-52face15d99f-kube-rbac-proxy-sar-config" pod "9228c263-f2fa-4e48-9dc4-52face15d99f" (UID: "9228c263-f2fa-4e48-9dc4-52face15d99f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:48.847842 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.847821    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9228c263-f2fa-4e48-9dc4-52face15d99f-kserve-provision-location" pod "9228c263-f2fa-4e48-9dc4-52face15d99f" (UID: "9228c263-f2fa-4e48-9dc4-52face15d99f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:48.848078 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.848052    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9228c263-f2fa-4e48-9dc4-52face15d99f-cabundle-cert" pod "9228c263-f2fa-4e48-9dc4-52face15d99f" (UID: "9228c263-f2fa-4e48-9dc4-52face15d99f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:48.849803 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.849776    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9228c263-f2fa-4e48-9dc4-52face15d99f-proxy-tls" pod "9228c263-f2fa-4e48-9dc4-52face15d99f" (UID: "9228c263-f2fa-4e48-9dc4-52face15d99f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:48.849936 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.849900    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9228c263-f2fa-4e48-9dc4-52face15d99f-kube-api-access-49gg5" pod "9228c263-f2fa-4e48-9dc4-52face15d99f" (UID: "9228c263-f2fa-4e48-9dc4-52face15d99f"). InnerVolumeSpecName "kube-api-access-49gg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:48.948645 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.948549    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9228c263-f2fa-4e48-9dc4-52face15d99f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:48.948645 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.948592    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9228c263-f2fa-4e48-9dc4-52face15d99f-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:48.948645 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.948602    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49gg5\" (UniqueName: \"kubernetes.io/projected/9228c263-f2fa-4e48-9dc4-52face15d99f-kube-api-access-49gg5\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:48.948645 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.948612    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9228c263-f2fa-4e48-9dc4-52face15d99f-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:48.948645 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:48.948621    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9228c263-f2fa-4e48-9dc4-52face15d99f-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:49.420516 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:49.420476    4473 generic.go:356] "Generic (PLEG): container finished" podID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerID="7966fa82ee786db69386f1ea472ca14f538fb18d18069ca76fc76206914f6ad5" exitCode=0
Oct 30 22:45:49.420693 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:49.420542    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"
Oct 30 22:45:49.420693 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:49.420565    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" event={"ID":"9228c263-f2fa-4e48-9dc4-52face15d99f","Type":"ContainerDied","Data":"7966fa82ee786db69386f1ea472ca14f538fb18d18069ca76fc76206914f6ad5"}
Oct 30 22:45:49.420693 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:49.420627    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg" event={"ID":"9228c263-f2fa-4e48-9dc4-52face15d99f","Type":"ContainerDied","Data":"3a369cf81d80f8200801ab708ee39cdd6ff2db910637a691f9b36d4c32ad95ce"}
Oct 30 22:45:49.420693 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:49.420653    4473 scope.go:122] "RemoveContainer" containerID="1466623da436525b0bbfdcdec08d617a5ff0bf570f62bffcd8b2e8e04f342d8c"
Oct 30 22:45:49.429166 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:49.429032    4473 scope.go:122] "RemoveContainer" containerID="7966fa82ee786db69386f1ea472ca14f538fb18d18069ca76fc76206914f6ad5"
Oct 30 22:45:49.436332 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:49.436313    4473 scope.go:122] "RemoveContainer" containerID="3c234e5078d7da762893aa331fcfcc539e91e25f7565115156c24c1126fedd70"
Oct 30 22:45:49.443544 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:49.443525    4473 scope.go:122] "RemoveContainer" containerID="1466623da436525b0bbfdcdec08d617a5ff0bf570f62bffcd8b2e8e04f342d8c"
Oct 30 22:45:49.443819 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:45:49.443800    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1466623da436525b0bbfdcdec08d617a5ff0bf570f62bffcd8b2e8e04f342d8c\": container with ID starting with 1466623da436525b0bbfdcdec08d617a5ff0bf570f62bffcd8b2e8e04f342d8c not found: ID does not exist" containerID="1466623da436525b0bbfdcdec08d617a5ff0bf570f62bffcd8b2e8e04f342d8c"
Oct 30 22:45:49.443857 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:49.443828    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1466623da436525b0bbfdcdec08d617a5ff0bf570f62bffcd8b2e8e04f342d8c"} err="failed to get container status \"1466623da436525b0bbfdcdec08d617a5ff0bf570f62bffcd8b2e8e04f342d8c\": rpc error: code = NotFound desc = could not find container \"1466623da436525b0bbfdcdec08d617a5ff0bf570f62bffcd8b2e8e04f342d8c\": container with ID starting with 1466623da436525b0bbfdcdec08d617a5ff0bf570f62bffcd8b2e8e04f342d8c not found: ID does not exist"
Oct 30 22:45:49.443857 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:49.443845    4473 scope.go:122] "RemoveContainer" containerID="7966fa82ee786db69386f1ea472ca14f538fb18d18069ca76fc76206914f6ad5"
Oct 30 22:45:49.444098 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:45:49.444083    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7966fa82ee786db69386f1ea472ca14f538fb18d18069ca76fc76206914f6ad5\": container with ID starting with 7966fa82ee786db69386f1ea472ca14f538fb18d18069ca76fc76206914f6ad5 not found: ID does not exist" containerID="7966fa82ee786db69386f1ea472ca14f538fb18d18069ca76fc76206914f6ad5"
Oct 30 22:45:49.444140 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:49.444104    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7966fa82ee786db69386f1ea472ca14f538fb18d18069ca76fc76206914f6ad5"} err="failed to get container status \"7966fa82ee786db69386f1ea472ca14f538fb18d18069ca76fc76206914f6ad5\": rpc error: code = NotFound desc = could not find container \"7966fa82ee786db69386f1ea472ca14f538fb18d18069ca76fc76206914f6ad5\": container with ID starting with 7966fa82ee786db69386f1ea472ca14f538fb18d18069ca76fc76206914f6ad5 not found: ID does not exist"
Oct 30 22:45:49.444140 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:49.444119    4473 scope.go:122] "RemoveContainer" containerID="3c234e5078d7da762893aa331fcfcc539e91e25f7565115156c24c1126fedd70"
Oct 30 22:45:49.444313 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:45:49.444297    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c234e5078d7da762893aa331fcfcc539e91e25f7565115156c24c1126fedd70\": container with ID starting with 3c234e5078d7da762893aa331fcfcc539e91e25f7565115156c24c1126fedd70 not found: ID does not exist" containerID="3c234e5078d7da762893aa331fcfcc539e91e25f7565115156c24c1126fedd70"
Oct 30 22:45:49.444345 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:49.444317    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c234e5078d7da762893aa331fcfcc539e91e25f7565115156c24c1126fedd70"} err="failed to get container status \"3c234e5078d7da762893aa331fcfcc539e91e25f7565115156c24c1126fedd70\": rpc error: code = NotFound desc = could not find container \"3c234e5078d7da762893aa331fcfcc539e91e25f7565115156c24c1126fedd70\": container with ID starting with 3c234e5078d7da762893aa331fcfcc539e91e25f7565115156c24c1126fedd70 not found: ID does not exist"
Oct 30 22:45:49.638848 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:49.638820    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"]
Oct 30 22:45:49.689210 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:49.689149    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-rollback-predictor-78cd59d4dc-wfpjg"]
Oct 30 22:45:49.886186 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:49.886153    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d521379-4169-4a34-89c0-2927c189a262" path="/var/lib/kubelet/pods/2d521379-4169-4a34-89c0-2927c189a262/volumes"
Oct 30 22:45:49.886626 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:49.886613    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" path="/var/lib/kubelet/pods/9228c263-f2fa-4e48-9dc4-52face15d99f/volumes"
Oct 30 22:45:50.426494 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:50.426466    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8_d7f1b7d9-c087-432e-bed8-5c0cf63d0eda/storage-initializer/0.log"
Oct 30 22:45:50.426692 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:50.426504    4473 generic.go:356] "Generic (PLEG): container finished" podID="d7f1b7d9-c087-432e-bed8-5c0cf63d0eda" containerID="e07cfdd09411b1070f31f883a4dd52aa0db8b7e089692db3dab6448ebb9c6fc0" exitCode=1
Oct 30 22:45:50.426692 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:50.426579    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8" event={"ID":"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda","Type":"ContainerDied","Data":"e07cfdd09411b1070f31f883a4dd52aa0db8b7e089692db3dab6448ebb9c6fc0"}
Oct 30 22:45:51.433662 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:51.433631    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8_d7f1b7d9-c087-432e-bed8-5c0cf63d0eda/storage-initializer/0.log"
Oct 30 22:45:51.434179 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:51.433758    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8" event={"ID":"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda","Type":"ContainerStarted","Data":"8c564751c3795a4a74164f8219191e14bed86d964272a6710a60f5aba0b11308"}
Oct 30 22:45:52.457793 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.457756    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:52.740405 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.740317    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:52.961652 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.961619    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"]
Oct 30 22:45:52.962098 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.962083    4473 cpu_manager.go:414] "containerMap: removing container" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kube-rbac-proxy"
Oct 30 22:45:52.962164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.962101    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kube-rbac-proxy"
Oct 30 22:45:52.962164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.962118    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="storage-initializer"
Oct 30 22:45:52.962164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.962126    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="storage-initializer"
Oct 30 22:45:52.962164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.962137    4473 cpu_manager.go:414] "containerMap: removing container" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kserve-container"
Oct 30 22:45:52.962164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.962144    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kserve-container"
Oct 30 22:45:52.962164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.962156    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kserve-container"
Oct 30 22:45:52.962164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.962163    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kserve-container"
Oct 30 22:45:52.962367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.962186    4473 cpu_manager.go:414] "containerMap: removing container" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="storage-initializer"
Oct 30 22:45:52.962367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.962193    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="storage-initializer"
Oct 30 22:45:52.962367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.962213    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kube-rbac-proxy"
Oct 30 22:45:52.962367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.962219    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kube-rbac-proxy"
Oct 30 22:45:52.962367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.962299    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kube-rbac-proxy"
Oct 30 22:45:52.962367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.962312    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kube-rbac-proxy"
Oct 30 22:45:52.962367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.962320    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d521379-4169-4a34-89c0-2927c189a262" containerName="kserve-container"
Oct 30 22:45:52.962367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.962328    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9228c263-f2fa-4e48-9dc4-52face15d99f" containerName="kserve-container"
Oct 30 22:45:52.965856 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.965838    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:52.994871 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:52.994803    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:53.019219 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.019189    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"default-dockercfg-4v4s8\""
Oct 30 22:45:53.069499 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.069471    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"kube-root-ca.crt\""
Oct 30 22:45:53.069499 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.069471    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:45:53.069718 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.069505    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Oct 30 22:45:53.069718 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.069514    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"openshift-service-ca.crt\""
Oct 30 22:45:53.069718 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.069531    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Oct 30 22:45:53.069718 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.069554    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"seaweedfs-s3-creds\""
Oct 30 22:45:53.071075 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.071060    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"]
Oct 30 22:45:53.083768 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.083725    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7fafd0e-2894-4f16-8772-1890eac3ae49-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-7677dd656b-dbpdf\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:53.083880 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.083791    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7fafd0e-2894-4f16-8772-1890eac3ae49-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-7677dd656b-dbpdf\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:53.083880 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.083863    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7fafd0e-2894-4f16-8772-1890eac3ae49-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-7677dd656b-dbpdf\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:53.084015 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.083895    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7fafd0e-2894-4f16-8772-1890eac3ae49-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-7677dd656b-dbpdf\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:53.084015 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.083919    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8hbx\" (UniqueName: \"kubernetes.io/projected/f7fafd0e-2894-4f16-8772-1890eac3ae49-kube-api-access-j8hbx\") pod \"isvc-canary-stop-v2-predictor-7677dd656b-dbpdf\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:53.185011 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.184979    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7fafd0e-2894-4f16-8772-1890eac3ae49-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-7677dd656b-dbpdf\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:53.185219 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.185026    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7fafd0e-2894-4f16-8772-1890eac3ae49-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-7677dd656b-dbpdf\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:53.185219 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.185061    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7fafd0e-2894-4f16-8772-1890eac3ae49-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-7677dd656b-dbpdf\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:53.185219 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.185089    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7fafd0e-2894-4f16-8772-1890eac3ae49-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-7677dd656b-dbpdf\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:53.185219 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.185123    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8hbx\" (UniqueName: \"kubernetes.io/projected/f7fafd0e-2894-4f16-8772-1890eac3ae49-kube-api-access-j8hbx\") pod \"isvc-canary-stop-v2-predictor-7677dd656b-dbpdf\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:53.185600 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.185574    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7fafd0e-2894-4f16-8772-1890eac3ae49-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-7677dd656b-dbpdf\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:53.185692 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.185613    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7fafd0e-2894-4f16-8772-1890eac3ae49-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-7677dd656b-dbpdf\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:53.185748 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.185721    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7fafd0e-2894-4f16-8772-1890eac3ae49-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-7677dd656b-dbpdf\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:53.187699 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.187679    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7fafd0e-2894-4f16-8772-1890eac3ae49-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-7677dd656b-dbpdf\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:53.244433 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.244403    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8hbx\" (UniqueName: \"kubernetes.io/projected/f7fafd0e-2894-4f16-8772-1890eac3ae49-kube-api-access-j8hbx\") pod \"isvc-canary-stop-v2-predictor-7677dd656b-dbpdf\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:53.288476 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.288394    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:53.306098 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.306069    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:53.493381 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.493352    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"]
Oct 30 22:45:53.497785 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.497769    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:53.550657 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.550589    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-canary-stop-predictor-serving-cert\""
Oct 30 22:45:53.556968 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.556928    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"]
Oct 30 22:45:53.588788 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.588755    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vqbv\" (UniqueName: \"kubernetes.io/projected/826fa7fe-6eea-476c-9e86-bfbac2c90704-kube-api-access-6vqbv\") pod \"isvc-canary-stop-predictor-748fd79575-8t599\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:53.588930 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.588800    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/826fa7fe-6eea-476c-9e86-bfbac2c90704-cabundle-cert\") pod \"isvc-canary-stop-predictor-748fd79575-8t599\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:53.588930 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.588821    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/826fa7fe-6eea-476c-9e86-bfbac2c90704-kserve-provision-location\") pod \"isvc-canary-stop-predictor-748fd79575-8t599\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:53.589051 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.588980    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/826fa7fe-6eea-476c-9e86-bfbac2c90704-proxy-tls\") pod \"isvc-canary-stop-predictor-748fd79575-8t599\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:53.589051 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.589025    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/826fa7fe-6eea-476c-9e86-bfbac2c90704-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-748fd79575-8t599\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:53.679838 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.679816    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"]
Oct 30 22:45:53.681785 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:45:53.681759    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fafd0e_2894_4f16_8772_1890eac3ae49.slice/crio-736291494e1ba3963df76f703f188dc1ddffbddd1996486269aa7772feda6e81 WatchSource:0}: Error finding container 736291494e1ba3963df76f703f188dc1ddffbddd1996486269aa7772feda6e81: Status 404 returned error can't find the container with id 736291494e1ba3963df76f703f188dc1ddffbddd1996486269aa7772feda6e81
Oct 30 22:45:53.689972 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.689932    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/826fa7fe-6eea-476c-9e86-bfbac2c90704-proxy-tls\") pod \"isvc-canary-stop-predictor-748fd79575-8t599\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:53.690076 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.690001    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/826fa7fe-6eea-476c-9e86-bfbac2c90704-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-748fd79575-8t599\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:53.690076 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.690037    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vqbv\" (UniqueName: \"kubernetes.io/projected/826fa7fe-6eea-476c-9e86-bfbac2c90704-kube-api-access-6vqbv\") pod \"isvc-canary-stop-predictor-748fd79575-8t599\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:53.690076 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.690069    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/826fa7fe-6eea-476c-9e86-bfbac2c90704-cabundle-cert\") pod \"isvc-canary-stop-predictor-748fd79575-8t599\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:53.690259 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.690095    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/826fa7fe-6eea-476c-9e86-bfbac2c90704-kserve-provision-location\") pod \"isvc-canary-stop-predictor-748fd79575-8t599\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:53.690413 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.690392    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/826fa7fe-6eea-476c-9e86-bfbac2c90704-kserve-provision-location\") pod \"isvc-canary-stop-predictor-748fd79575-8t599\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:53.690814 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.690792    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/826fa7fe-6eea-476c-9e86-bfbac2c90704-cabundle-cert\") pod \"isvc-canary-stop-predictor-748fd79575-8t599\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:53.690866 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.690792    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/826fa7fe-6eea-476c-9e86-bfbac2c90704-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-748fd79575-8t599\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:53.692582 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.692554    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/826fa7fe-6eea-476c-9e86-bfbac2c90704-proxy-tls\") pod \"isvc-canary-stop-predictor-748fd79575-8t599\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:53.747342 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.747321    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vqbv\" (UniqueName: \"kubernetes.io/projected/826fa7fe-6eea-476c-9e86-bfbac2c90704-kube-api-access-6vqbv\") pod \"isvc-canary-stop-predictor-748fd79575-8t599\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") " pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:53.814790 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:53.814708    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:54.189378 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:54.189351    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"]
Oct 30 22:45:54.191659 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:45:54.191629    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826fa7fe_6eea_476c_9e86_bfbac2c90704.slice/crio-e25158aad1d1c621eff19d89589503a5d0cf1832759e3c1f3ca63f38e86304ef WatchSource:0}: Error finding container e25158aad1d1c621eff19d89589503a5d0cf1832759e3c1f3ca63f38e86304ef: Status 404 returned error can't find the container with id e25158aad1d1c621eff19d89589503a5d0cf1832759e3c1f3ca63f38e86304ef
Oct 30 22:45:54.449211 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:54.449108    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" event={"ID":"f7fafd0e-2894-4f16-8772-1890eac3ae49","Type":"ContainerStarted","Data":"4d943f44c70d0861dcd6901b2fefa34dc60328d6a68cbb5591f05c4604363379"}
Oct 30 22:45:54.449211 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:54.449181    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" event={"ID":"f7fafd0e-2894-4f16-8772-1890eac3ae49","Type":"ContainerStarted","Data":"736291494e1ba3963df76f703f188dc1ddffbddd1996486269aa7772feda6e81"}
Oct 30 22:45:54.450697 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:54.450667    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" event={"ID":"826fa7fe-6eea-476c-9e86-bfbac2c90704","Type":"ContainerStarted","Data":"68b770502a28a40269416fb14c6f134f54caad46f3287bb6670fc1db8925e7d3"}
Oct 30 22:45:54.450802 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:54.450725    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" event={"ID":"826fa7fe-6eea-476c-9e86-bfbac2c90704","Type":"ContainerStarted","Data":"e25158aad1d1c621eff19d89589503a5d0cf1832759e3c1f3ca63f38e86304ef"}
Oct 30 22:45:55.774623 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:55.774584    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"]
Oct 30 22:45:55.775065 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:55.774852    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8" podUID="d7f1b7d9-c087-432e-bed8-5c0cf63d0eda" containerName="storage-initializer" containerID="cri-o://8c564751c3795a4a74164f8219191e14bed86d964272a6710a60f5aba0b11308" gracePeriod=30
Oct 30 22:45:55.807679 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:55.807652    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:57.116091 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.116067    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8_d7f1b7d9-c087-432e-bed8-5c0cf63d0eda/storage-initializer/1.log"
Oct 30 22:45:57.116503 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.116489    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8_d7f1b7d9-c087-432e-bed8-5c0cf63d0eda/storage-initializer/0.log"
Oct 30 22:45:57.118884 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.118869    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"
Oct 30 22:45:57.125839 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.125823    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kube-api-access-pmng7\" (UniqueName: \"kubernetes.io/projected/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kube-api-access-pmng7\") pod \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\" (UID: \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\") "
Oct 30 22:45:57.125898 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.125869    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kube-rbac-proxy-sar-config\") pod \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\" (UID: \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\") "
Oct 30 22:45:57.125940 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.125903    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kserve-provision-location\") pod \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\" (UID: \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\") "
Oct 30 22:45:57.126034 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.125968    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-proxy-tls\") pod \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\" (UID: \"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda\") "
Oct 30 22:45:57.126192 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.126172    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kserve-provision-location" pod "d7f1b7d9-c087-432e-bed8-5c0cf63d0eda" (UID: "d7f1b7d9-c087-432e-bed8-5c0cf63d0eda"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:57.126284 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.126261    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kube-rbac-proxy-sar-config" pod "d7f1b7d9-c087-432e-bed8-5c0cf63d0eda" (UID: "d7f1b7d9-c087-432e-bed8-5c0cf63d0eda"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:57.127920 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.127899    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kube-api-access-pmng7" pod "d7f1b7d9-c087-432e-bed8-5c0cf63d0eda" (UID: "d7f1b7d9-c087-432e-bed8-5c0cf63d0eda"). InnerVolumeSpecName "kube-api-access-pmng7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:57.128074 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.128057    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-proxy-tls" pod "d7f1b7d9-c087-432e-bed8-5c0cf63d0eda" (UID: "d7f1b7d9-c087-432e-bed8-5c0cf63d0eda"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:57.227176 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.227104    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:57.227176 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.227131    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmng7\" (UniqueName: \"kubernetes.io/projected/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kube-api-access-pmng7\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:57.227176 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.227142    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:57.227176 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.227151    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:57.465222 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.465187    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8_d7f1b7d9-c087-432e-bed8-5c0cf63d0eda/storage-initializer/1.log"
Oct 30 22:45:57.465572 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.465557    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8_d7f1b7d9-c087-432e-bed8-5c0cf63d0eda/storage-initializer/0.log"
Oct 30 22:45:57.465635 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.465589    4473 generic.go:356] "Generic (PLEG): container finished" podID="d7f1b7d9-c087-432e-bed8-5c0cf63d0eda" containerID="8c564751c3795a4a74164f8219191e14bed86d964272a6710a60f5aba0b11308" exitCode=1
Oct 30 22:45:57.465692 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.465626    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8" event={"ID":"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda","Type":"ContainerDied","Data":"8c564751c3795a4a74164f8219191e14bed86d964272a6710a60f5aba0b11308"}
Oct 30 22:45:57.465692 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.465646    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"
Oct 30 22:45:57.465805 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.465709    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8" event={"ID":"d7f1b7d9-c087-432e-bed8-5c0cf63d0eda","Type":"ContainerDied","Data":"126555ae8097510052bfd42942604dcdfe1055049368f75872021a805ba0cd56"}
Oct 30 22:45:57.465805 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.465754    4473 scope.go:122] "RemoveContainer" containerID="8c564751c3795a4a74164f8219191e14bed86d964272a6710a60f5aba0b11308"
Oct 30 22:45:57.474530 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.474512    4473 scope.go:122] "RemoveContainer" containerID="e07cfdd09411b1070f31f883a4dd52aa0db8b7e089692db3dab6448ebb9c6fc0"
Oct 30 22:45:57.481808 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.481794    4473 scope.go:122] "RemoveContainer" containerID="8c564751c3795a4a74164f8219191e14bed86d964272a6710a60f5aba0b11308"
Oct 30 22:45:57.482092 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:45:57.482074    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c564751c3795a4a74164f8219191e14bed86d964272a6710a60f5aba0b11308\": container with ID starting with 8c564751c3795a4a74164f8219191e14bed86d964272a6710a60f5aba0b11308 not found: ID does not exist" containerID="8c564751c3795a4a74164f8219191e14bed86d964272a6710a60f5aba0b11308"
Oct 30 22:45:57.482146 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.482101    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c564751c3795a4a74164f8219191e14bed86d964272a6710a60f5aba0b11308"} err="failed to get container status \"8c564751c3795a4a74164f8219191e14bed86d964272a6710a60f5aba0b11308\": rpc error: code = NotFound desc = could not find container \"8c564751c3795a4a74164f8219191e14bed86d964272a6710a60f5aba0b11308\": container with ID starting with 8c564751c3795a4a74164f8219191e14bed86d964272a6710a60f5aba0b11308 not found: ID does not exist"
Oct 30 22:45:57.482146 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.482120    4473 scope.go:122] "RemoveContainer" containerID="e07cfdd09411b1070f31f883a4dd52aa0db8b7e089692db3dab6448ebb9c6fc0"
Oct 30 22:45:57.482347 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:45:57.482328    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07cfdd09411b1070f31f883a4dd52aa0db8b7e089692db3dab6448ebb9c6fc0\": container with ID starting with e07cfdd09411b1070f31f883a4dd52aa0db8b7e089692db3dab6448ebb9c6fc0 not found: ID does not exist" containerID="e07cfdd09411b1070f31f883a4dd52aa0db8b7e089692db3dab6448ebb9c6fc0"
Oct 30 22:45:57.482387 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.482353    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07cfdd09411b1070f31f883a4dd52aa0db8b7e089692db3dab6448ebb9c6fc0"} err="failed to get container status \"e07cfdd09411b1070f31f883a4dd52aa0db8b7e089692db3dab6448ebb9c6fc0\": rpc error: code = NotFound desc = could not find container \"e07cfdd09411b1070f31f883a4dd52aa0db8b7e089692db3dab6448ebb9c6fc0\": container with ID starting with e07cfdd09411b1070f31f883a4dd52aa0db8b7e089692db3dab6448ebb9c6fc0 not found: ID does not exist"
Oct 30 22:45:57.698835 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.698801    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"]
Oct 30 22:45:57.704380 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.704352    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-global-fail-predictor-f86bb78b8-dqqw8"]
Oct 30 22:45:57.886997 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:57.886905    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f1b7d9-c087-432e-bed8-5c0cf63d0eda" path="/var/lib/kubelet/pods/d7f1b7d9-c087-432e-bed8-5c0cf63d0eda/volumes"
Oct 30 22:45:58.471669 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:58.471636    4473 generic.go:356] "Generic (PLEG): container finished" podID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerID="68b770502a28a40269416fb14c6f134f54caad46f3287bb6670fc1db8925e7d3" exitCode=0
Oct 30 22:45:58.472153 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:58.471714    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" event={"ID":"826fa7fe-6eea-476c-9e86-bfbac2c90704","Type":"ContainerDied","Data":"68b770502a28a40269416fb14c6f134f54caad46f3287bb6670fc1db8925e7d3"}
Oct 30 22:45:58.473623 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:58.473526    4473 generic.go:356] "Generic (PLEG): container finished" podID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerID="4d943f44c70d0861dcd6901b2fefa34dc60328d6a68cbb5591f05c4604363379" exitCode=0
Oct 30 22:45:58.473623 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:58.473597    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" event={"ID":"f7fafd0e-2894-4f16-8772-1890eac3ae49","Type":"ContainerDied","Data":"4d943f44c70d0861dcd6901b2fefa34dc60328d6a68cbb5591f05c4604363379"}
Oct 30 22:45:59.480926 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:59.480884    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" event={"ID":"826fa7fe-6eea-476c-9e86-bfbac2c90704","Type":"ContainerStarted","Data":"86b82fd1d5a5c747e3896e738c582a43db17803b3d5ef42a950b081faa464d38"}
Oct 30 22:45:59.481391 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:59.480990    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" event={"ID":"826fa7fe-6eea-476c-9e86-bfbac2c90704","Type":"ContainerStarted","Data":"32f92e1540a94d6dcc75b8c91f3db2784fce6288009835d7a72a1a5c3a4ce6f4"}
Oct 30 22:45:59.481391 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:59.481301    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:45:59.483354 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:59.483321    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" event={"ID":"f7fafd0e-2894-4f16-8772-1890eac3ae49","Type":"ContainerStarted","Data":"2554a5d89d0b0c91bd74de5b8f76243cd99029d592d49e47efe9f1b4b0ee6ea8"}
Oct 30 22:45:59.483470 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:59.483378    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" event={"ID":"f7fafd0e-2894-4f16-8772-1890eac3ae49","Type":"ContainerStarted","Data":"40ad8bf5a482e7954aaeeb49f4927b889f68acbb1efbfd2c0ffe96d1210ef600"}
Oct 30 22:45:59.483655 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:59.483629    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:59.483780 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:59.483680    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:45:59.484811 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:59.484790    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Oct 30 22:45:59.601497 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:59.601444    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" podStartSLOduration=2.807253944 podStartE2EDuration="6.601430276s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.794176332s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:59.600768775 +0000 UTC m=+3177.836978121" watchObservedRunningTime="2026-10-30 22:45:59.601430276 +0000 UTC m=+3177.837639620"
Oct 30 22:45:59.721675 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:45:59.721625    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" podStartSLOduration=3.5013480980000002 podStartE2EDuration="7.721608545s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.220260447s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:59.720814644 +0000 UTC m=+3177.957023988" watchObservedRunningTime="2026-10-30 22:45:59.721608545 +0000 UTC m=+3177.957817891"
Oct 30 22:46:00.487794 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:00.487749    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Oct 30 22:46:00.487794 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:00.487793    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:46:00.488940 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:00.488919    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Oct 30 22:46:01.491967 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:01.491915    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Oct 30 22:46:02.214820 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.214786    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:02.583901 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.583825    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"]
Oct 30 22:46:02.584305 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.584210    4473 cpu_manager.go:414] "containerMap: removing container" podUID="d7f1b7d9-c087-432e-bed8-5c0cf63d0eda" containerName="storage-initializer"
Oct 30 22:46:02.584305 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.584221    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f1b7d9-c087-432e-bed8-5c0cf63d0eda" containerName="storage-initializer"
Oct 30 22:46:02.584305 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.584231    4473 cpu_manager.go:414] "containerMap: removing container" podUID="d7f1b7d9-c087-432e-bed8-5c0cf63d0eda" containerName="storage-initializer"
Oct 30 22:46:02.584305 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.584236    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f1b7d9-c087-432e-bed8-5c0cf63d0eda" containerName="storage-initializer"
Oct 30 22:46:02.584305 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.584287    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f1b7d9-c087-432e-bed8-5c0cf63d0eda" containerName="storage-initializer"
Oct 30 22:46:02.584305 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.584296    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f1b7d9-c087-432e-bed8-5c0cf63d0eda" containerName="storage-initializer"
Oct 30 22:46:02.586877 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.586846    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:02.588575 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.588561    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:02.644461 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.644425    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:46:02.672867 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.672837    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vpvg\" (UniqueName: \"kubernetes.io/projected/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kube-api-access-9vpvg\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:02.673048 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.672876    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:02.673048 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.672915    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fce83cb9-ddcb-4ce7-abed-7df75e01b841-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:02.673126 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.673090    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:02.673126 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.673121    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fce83cb9-ddcb-4ce7-abed-7df75e01b841-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:02.694499 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.694469    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Oct 30 22:46:02.694499 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.694479    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:46:02.694681 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.694483    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"storage-config\""
Oct 30 22:46:02.694681 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.694479    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:46:02.694681 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.694623    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Oct 30 22:46:02.694681 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.694655    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:46:02.696540 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.696522    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"]
Oct 30 22:46:02.773509 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.773478    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fce83cb9-ddcb-4ce7-abed-7df75e01b841-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:02.773678 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.773570    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:02.773678 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.773589    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fce83cb9-ddcb-4ce7-abed-7df75e01b841-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:02.773678 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.773620    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vpvg\" (UniqueName: \"kubernetes.io/projected/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kube-api-access-9vpvg\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:02.773678 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.773636    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:02.774057 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.774039    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:02.774230 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.774213    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:02.774266 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.774249    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fce83cb9-ddcb-4ce7-abed-7df75e01b841-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:02.776270 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.776251    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fce83cb9-ddcb-4ce7-abed-7df75e01b841-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:02.832726 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.832693    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vpvg\" (UniqueName: \"kubernetes.io/projected/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kube-api-access-9vpvg\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:02.905617 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:02.905589    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:03.344876 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:03.344657    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"]
Oct 30 22:46:03.347351 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:46:03.347309    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce83cb9_ddcb_4ce7_abed_7df75e01b841.slice/crio-c2ffe723298abe23214e1ffc0644451815fed75c6f5077bc7e7d2d89698826ab WatchSource:0}: Error finding container c2ffe723298abe23214e1ffc0644451815fed75c6f5077bc7e7d2d89698826ab: Status 404 returned error can't find the container with id c2ffe723298abe23214e1ffc0644451815fed75c6f5077bc7e7d2d89698826ab
Oct 30 22:46:03.502681 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:03.502646    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" event={"ID":"fce83cb9-ddcb-4ce7-abed-7df75e01b841","Type":"ContainerStarted","Data":"20571980f3837a8170b0467d32c7bb65c9822b6f045253fa526c8c62c35d090b"}
Oct 30 22:46:03.502833 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:03.502703    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" event={"ID":"fce83cb9-ddcb-4ce7-abed-7df75e01b841","Type":"ContainerStarted","Data":"c2ffe723298abe23214e1ffc0644451815fed75c6f5077bc7e7d2d89698826ab"}
Oct 30 22:46:04.508130 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:04.508092    4473 generic.go:356] "Generic (PLEG): container finished" podID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerID="20571980f3837a8170b0467d32c7bb65c9822b6f045253fa526c8c62c35d090b" exitCode=0
Oct 30 22:46:04.508497 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:04.508158    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" event={"ID":"fce83cb9-ddcb-4ce7-abed-7df75e01b841","Type":"ContainerDied","Data":"20571980f3837a8170b0467d32c7bb65c9822b6f045253fa526c8c62c35d090b"}
Oct 30 22:46:05.493035 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:05.493009    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:46:05.493509 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:05.493475    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Oct 30 22:46:05.514906 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:05.514878    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" event={"ID":"fce83cb9-ddcb-4ce7-abed-7df75e01b841","Type":"ContainerStarted","Data":"27514699c8e7f45671e97a56f1ffcaa752c85e2ed5814aa95eee81b7986c62dd"}
Oct 30 22:46:05.515262 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:05.514934    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" event={"ID":"fce83cb9-ddcb-4ce7-abed-7df75e01b841","Type":"ContainerStarted","Data":"f866e84eb8e0383fce36c9a02d9e14ded5676ae513a9cba5d41390318cbae295"}
Oct 30 22:46:05.515262 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:05.514977    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:05.515262 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:05.515000    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:05.516154 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:05.516134    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Oct 30 22:46:05.783295 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:05.783191    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" podStartSLOduration=2.7539743469999998 podStartE2EDuration="3.783175195s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.029200848s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:05.78259774 +0000 UTC m=+3184.018807087" watchObservedRunningTime="2026-10-30 22:46:05.783175195 +0000 UTC m=+3184.019384539"
Oct 30 22:46:06.496640 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:06.496611    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:46:06.497109 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:06.497083    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Oct 30 22:46:06.519177 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:06.519137    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Oct 30 22:46:09.189276 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:09.189239    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:11.303752 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:46:11.303719    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:46:11.523210 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:11.523182    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:46:11.523676 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:11.523653    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Oct 30 22:46:15.493759 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:15.493680    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Oct 30 22:46:16.497544 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:16.497504    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Oct 30 22:46:21.524324 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:21.524282    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Oct 30 22:46:25.494212 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:25.494169    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Oct 30 22:46:26.497339 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:26.497295    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Oct 30 22:46:31.524112 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:31.524071    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Oct 30 22:46:35.493858 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:35.493817    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Oct 30 22:46:36.497148 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:36.497108    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Oct 30 22:46:41.303361 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:46:41.303324    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:46:41.524271 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:41.524233    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Oct 30 22:46:45.493540 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:45.493501    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Oct 30 22:46:46.497285 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:46.497247    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Oct 30 22:46:51.524606 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:51.524567    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Oct 30 22:46:55.493588 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:55.493549    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Oct 30 22:46:56.497705 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:46:56.497666    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Oct 30 22:47:01.524467 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:01.524424    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Oct 30 22:47:05.494340 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:05.494297    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Oct 30 22:47:06.497654 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:06.497614    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Oct 30 22:47:10.711078 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:10.711037    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:11.303170 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:47:11.303136    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:47:11.523850 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:11.523810    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Oct 30 22:47:15.494005 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:15.493969    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Oct 30 22:47:16.498261 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:16.498233    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:47:20.882106 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:20.882070    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:47:21.523658 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:21.523617    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Oct 30 22:47:22.282998 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:22.282965    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"]
Oct 30 22:47:22.283450 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:22.283193    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kserve-container" containerID="cri-o://32f92e1540a94d6dcc75b8c91f3db2784fce6288009835d7a72a1a5c3a4ce6f4" gracePeriod=30
Oct 30 22:47:22.283450 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:22.283217    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kube-rbac-proxy" containerID="cri-o://86b82fd1d5a5c747e3896e738c582a43db17803b3d5ef42a950b081faa464d38" gracePeriod=30
Oct 30 22:47:22.850621 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:22.850588    4473 generic.go:356] "Generic (PLEG): container finished" podID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerID="86b82fd1d5a5c747e3896e738c582a43db17803b3d5ef42a950b081faa464d38" exitCode=0
Oct 30 22:47:22.850815 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:22.850667    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" event={"ID":"826fa7fe-6eea-476c-9e86-bfbac2c90704","Type":"ContainerDied","Data":"86b82fd1d5a5c747e3896e738c582a43db17803b3d5ef42a950b081faa464d38"}
Oct 30 22:47:23.122488 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:23.122378    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:23.172758 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:23.172724    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"]
Oct 30 22:47:23.173011 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:23.172982    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kserve-container" containerID="cri-o://40ad8bf5a482e7954aaeeb49f4927b889f68acbb1efbfd2c0ffe96d1210ef600" gracePeriod=30
Oct 30 22:47:23.173173 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:23.173024    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kube-rbac-proxy" containerID="cri-o://2554a5d89d0b0c91bd74de5b8f76243cd99029d592d49e47efe9f1b4b0ee6ea8" gracePeriod=30
Oct 30 22:47:23.654526 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:23.654496    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:23.857319 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:23.857282    4473 generic.go:356] "Generic (PLEG): container finished" podID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerID="2554a5d89d0b0c91bd74de5b8f76243cd99029d592d49e47efe9f1b4b0ee6ea8" exitCode=0
Oct 30 22:47:23.857498 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:23.857355    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" event={"ID":"f7fafd0e-2894-4f16-8772-1890eac3ae49","Type":"ContainerDied","Data":"2554a5d89d0b0c91bd74de5b8f76243cd99029d592d49e47efe9f1b4b0ee6ea8"}
Oct 30 22:47:25.488291 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:25.488254    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.67:8643/healthz\": dial tcp 10.134.0.67:8643: connect: connection refused"
Oct 30 22:47:26.493197 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:26.493157    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.68:8643/healthz\": dial tcp 10.134.0.68:8643: connect: connection refused"
Oct 30 22:47:26.497511 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:26.497490    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Oct 30 22:47:27.331123 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.331099    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:47:27.486940 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.486858    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826fa7fe-6eea-476c-9e86-bfbac2c90704-kube-api-access-6vqbv\" (UniqueName: \"kubernetes.io/projected/826fa7fe-6eea-476c-9e86-bfbac2c90704-kube-api-access-6vqbv\") pod \"826fa7fe-6eea-476c-9e86-bfbac2c90704\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") "
Oct 30 22:47:27.487103 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.487013    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826fa7fe-6eea-476c-9e86-bfbac2c90704-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/826fa7fe-6eea-476c-9e86-bfbac2c90704-cabundle-cert\") pod \"826fa7fe-6eea-476c-9e86-bfbac2c90704\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") "
Oct 30 22:47:27.487103 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.487049    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826fa7fe-6eea-476c-9e86-bfbac2c90704-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/826fa7fe-6eea-476c-9e86-bfbac2c90704-kserve-provision-location\") pod \"826fa7fe-6eea-476c-9e86-bfbac2c90704\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") "
Oct 30 22:47:27.487103 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.487076    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/826fa7fe-6eea-476c-9e86-bfbac2c90704-proxy-tls\" (UniqueName: \"kubernetes.io/secret/826fa7fe-6eea-476c-9e86-bfbac2c90704-proxy-tls\") pod \"826fa7fe-6eea-476c-9e86-bfbac2c90704\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") "
Oct 30 22:47:27.487269 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.487103    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826fa7fe-6eea-476c-9e86-bfbac2c90704-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/826fa7fe-6eea-476c-9e86-bfbac2c90704-kube-rbac-proxy-sar-config\") pod \"826fa7fe-6eea-476c-9e86-bfbac2c90704\" (UID: \"826fa7fe-6eea-476c-9e86-bfbac2c90704\") "
Oct 30 22:47:27.487448 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.487358    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826fa7fe-6eea-476c-9e86-bfbac2c90704-kserve-provision-location" pod "826fa7fe-6eea-476c-9e86-bfbac2c90704" (UID: "826fa7fe-6eea-476c-9e86-bfbac2c90704"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:27.487448 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.487438    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826fa7fe-6eea-476c-9e86-bfbac2c90704-cabundle-cert" pod "826fa7fe-6eea-476c-9e86-bfbac2c90704" (UID: "826fa7fe-6eea-476c-9e86-bfbac2c90704"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:27.487572 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.487487    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826fa7fe-6eea-476c-9e86-bfbac2c90704-kube-rbac-proxy-sar-config" pod "826fa7fe-6eea-476c-9e86-bfbac2c90704" (UID: "826fa7fe-6eea-476c-9e86-bfbac2c90704"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:27.489197 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.489175    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826fa7fe-6eea-476c-9e86-bfbac2c90704-proxy-tls" pod "826fa7fe-6eea-476c-9e86-bfbac2c90704" (UID: "826fa7fe-6eea-476c-9e86-bfbac2c90704"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:27.489256 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.489243    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826fa7fe-6eea-476c-9e86-bfbac2c90704-kube-api-access-6vqbv" pod "826fa7fe-6eea-476c-9e86-bfbac2c90704" (UID: "826fa7fe-6eea-476c-9e86-bfbac2c90704"). InnerVolumeSpecName "kube-api-access-6vqbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:27.588407 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.588371    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/826fa7fe-6eea-476c-9e86-bfbac2c90704-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:27.588407 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.588402    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/826fa7fe-6eea-476c-9e86-bfbac2c90704-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:27.588407 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.588413    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/826fa7fe-6eea-476c-9e86-bfbac2c90704-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:27.588852 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.588423    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/826fa7fe-6eea-476c-9e86-bfbac2c90704-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:27.588852 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.588432    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vqbv\" (UniqueName: \"kubernetes.io/projected/826fa7fe-6eea-476c-9e86-bfbac2c90704-kube-api-access-6vqbv\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:27.876505 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.876423    4473 generic.go:356] "Generic (PLEG): container finished" podID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerID="32f92e1540a94d6dcc75b8c91f3db2784fce6288009835d7a72a1a5c3a4ce6f4" exitCode=0
Oct 30 22:47:27.876505 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.876486    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"
Oct 30 22:47:27.876709 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.876504    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" event={"ID":"826fa7fe-6eea-476c-9e86-bfbac2c90704","Type":"ContainerDied","Data":"32f92e1540a94d6dcc75b8c91f3db2784fce6288009835d7a72a1a5c3a4ce6f4"}
Oct 30 22:47:27.876709 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.876571    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599" event={"ID":"826fa7fe-6eea-476c-9e86-bfbac2c90704","Type":"ContainerDied","Data":"e25158aad1d1c621eff19d89589503a5d0cf1832759e3c1f3ca63f38e86304ef"}
Oct 30 22:47:27.876709 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.876601    4473 scope.go:122] "RemoveContainer" containerID="86b82fd1d5a5c747e3896e738c582a43db17803b3d5ef42a950b081faa464d38"
Oct 30 22:47:27.885814 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.885799    4473 scope.go:122] "RemoveContainer" containerID="32f92e1540a94d6dcc75b8c91f3db2784fce6288009835d7a72a1a5c3a4ce6f4"
Oct 30 22:47:27.893674 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.893652    4473 scope.go:122] "RemoveContainer" containerID="68b770502a28a40269416fb14c6f134f54caad46f3287bb6670fc1db8925e7d3"
Oct 30 22:47:27.901486 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.901466    4473 scope.go:122] "RemoveContainer" containerID="86b82fd1d5a5c747e3896e738c582a43db17803b3d5ef42a950b081faa464d38"
Oct 30 22:47:27.901745 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:47:27.901722    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b82fd1d5a5c747e3896e738c582a43db17803b3d5ef42a950b081faa464d38\": container with ID starting with 86b82fd1d5a5c747e3896e738c582a43db17803b3d5ef42a950b081faa464d38 not found: ID does not exist" containerID="86b82fd1d5a5c747e3896e738c582a43db17803b3d5ef42a950b081faa464d38"
Oct 30 22:47:27.901795 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.901751    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b82fd1d5a5c747e3896e738c582a43db17803b3d5ef42a950b081faa464d38"} err="failed to get container status \"86b82fd1d5a5c747e3896e738c582a43db17803b3d5ef42a950b081faa464d38\": rpc error: code = NotFound desc = could not find container \"86b82fd1d5a5c747e3896e738c582a43db17803b3d5ef42a950b081faa464d38\": container with ID starting with 86b82fd1d5a5c747e3896e738c582a43db17803b3d5ef42a950b081faa464d38 not found: ID does not exist"
Oct 30 22:47:27.901795 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.901767    4473 scope.go:122] "RemoveContainer" containerID="32f92e1540a94d6dcc75b8c91f3db2784fce6288009835d7a72a1a5c3a4ce6f4"
Oct 30 22:47:27.902020 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:47:27.902005    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f92e1540a94d6dcc75b8c91f3db2784fce6288009835d7a72a1a5c3a4ce6f4\": container with ID starting with 32f92e1540a94d6dcc75b8c91f3db2784fce6288009835d7a72a1a5c3a4ce6f4 not found: ID does not exist" containerID="32f92e1540a94d6dcc75b8c91f3db2784fce6288009835d7a72a1a5c3a4ce6f4"
Oct 30 22:47:27.902060 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.902025    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f92e1540a94d6dcc75b8c91f3db2784fce6288009835d7a72a1a5c3a4ce6f4"} err="failed to get container status \"32f92e1540a94d6dcc75b8c91f3db2784fce6288009835d7a72a1a5c3a4ce6f4\": rpc error: code = NotFound desc = could not find container \"32f92e1540a94d6dcc75b8c91f3db2784fce6288009835d7a72a1a5c3a4ce6f4\": container with ID starting with 32f92e1540a94d6dcc75b8c91f3db2784fce6288009835d7a72a1a5c3a4ce6f4 not found: ID does not exist"
Oct 30 22:47:27.902060 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.902042    4473 scope.go:122] "RemoveContainer" containerID="68b770502a28a40269416fb14c6f134f54caad46f3287bb6670fc1db8925e7d3"
Oct 30 22:47:27.902264 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:47:27.902249    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b770502a28a40269416fb14c6f134f54caad46f3287bb6670fc1db8925e7d3\": container with ID starting with 68b770502a28a40269416fb14c6f134f54caad46f3287bb6670fc1db8925e7d3 not found: ID does not exist" containerID="68b770502a28a40269416fb14c6f134f54caad46f3287bb6670fc1db8925e7d3"
Oct 30 22:47:27.902299 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:27.902270    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b770502a28a40269416fb14c6f134f54caad46f3287bb6670fc1db8925e7d3"} err="failed to get container status \"68b770502a28a40269416fb14c6f134f54caad46f3287bb6670fc1db8925e7d3\": rpc error: code = NotFound desc = could not find container \"68b770502a28a40269416fb14c6f134f54caad46f3287bb6670fc1db8925e7d3\": container with ID starting with 68b770502a28a40269416fb14c6f134f54caad46f3287bb6670fc1db8925e7d3 not found: ID does not exist"
Oct 30 22:47:28.044917 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.044882    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"]
Oct 30 22:47:28.046519 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.046499    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-stop-predictor-748fd79575-8t599"]
Oct 30 22:47:28.222756 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.222733    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:47:28.294269 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.294235    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7fafd0e-2894-4f16-8772-1890eac3ae49-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7fafd0e-2894-4f16-8772-1890eac3ae49-cabundle-cert\") pod \"f7fafd0e-2894-4f16-8772-1890eac3ae49\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") "
Oct 30 22:47:28.294447 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.294282    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7fafd0e-2894-4f16-8772-1890eac3ae49-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7fafd0e-2894-4f16-8772-1890eac3ae49-kube-rbac-proxy-sar-config\") pod \"f7fafd0e-2894-4f16-8772-1890eac3ae49\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") "
Oct 30 22:47:28.294447 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.294409    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7fafd0e-2894-4f16-8772-1890eac3ae49-proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7fafd0e-2894-4f16-8772-1890eac3ae49-proxy-tls\") pod \"f7fafd0e-2894-4f16-8772-1890eac3ae49\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") "
Oct 30 22:47:28.294556 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.294445    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7fafd0e-2894-4f16-8772-1890eac3ae49-kube-api-access-j8hbx\" (UniqueName: \"kubernetes.io/projected/f7fafd0e-2894-4f16-8772-1890eac3ae49-kube-api-access-j8hbx\") pod \"f7fafd0e-2894-4f16-8772-1890eac3ae49\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") "
Oct 30 22:47:28.294556 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.294476    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7fafd0e-2894-4f16-8772-1890eac3ae49-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7fafd0e-2894-4f16-8772-1890eac3ae49-kserve-provision-location\") pod \"f7fafd0e-2894-4f16-8772-1890eac3ae49\" (UID: \"f7fafd0e-2894-4f16-8772-1890eac3ae49\") "
Oct 30 22:47:28.294680 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.294659    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7fafd0e-2894-4f16-8772-1890eac3ae49-cabundle-cert" pod "f7fafd0e-2894-4f16-8772-1890eac3ae49" (UID: "f7fafd0e-2894-4f16-8772-1890eac3ae49"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:28.294748 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.294667    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7fafd0e-2894-4f16-8772-1890eac3ae49-kube-rbac-proxy-sar-config" pod "f7fafd0e-2894-4f16-8772-1890eac3ae49" (UID: "f7fafd0e-2894-4f16-8772-1890eac3ae49"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:28.294883 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.294863    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7fafd0e-2894-4f16-8772-1890eac3ae49-kserve-provision-location" pod "f7fafd0e-2894-4f16-8772-1890eac3ae49" (UID: "f7fafd0e-2894-4f16-8772-1890eac3ae49"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:28.296794 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.296768    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7fafd0e-2894-4f16-8772-1890eac3ae49-proxy-tls" pod "f7fafd0e-2894-4f16-8772-1890eac3ae49" (UID: "f7fafd0e-2894-4f16-8772-1890eac3ae49"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:28.296915 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.296809    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7fafd0e-2894-4f16-8772-1890eac3ae49-kube-api-access-j8hbx" pod "f7fafd0e-2894-4f16-8772-1890eac3ae49" (UID: "f7fafd0e-2894-4f16-8772-1890eac3ae49"). InnerVolumeSpecName "kube-api-access-j8hbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:28.395531 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.395499    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7fafd0e-2894-4f16-8772-1890eac3ae49-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:28.395531 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.395529    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8hbx\" (UniqueName: \"kubernetes.io/projected/f7fafd0e-2894-4f16-8772-1890eac3ae49-kube-api-access-j8hbx\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:28.395747 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.395539    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7fafd0e-2894-4f16-8772-1890eac3ae49-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:28.395747 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.395550    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7fafd0e-2894-4f16-8772-1890eac3ae49-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:28.395747 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.395559    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7fafd0e-2894-4f16-8772-1890eac3ae49-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:28.883230 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.883191    4473 generic.go:356] "Generic (PLEG): container finished" podID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerID="40ad8bf5a482e7954aaeeb49f4927b889f68acbb1efbfd2c0ffe96d1210ef600" exitCode=0
Oct 30 22:47:28.883700 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.883247    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"
Oct 30 22:47:28.883700 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.883274    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" event={"ID":"f7fafd0e-2894-4f16-8772-1890eac3ae49","Type":"ContainerDied","Data":"40ad8bf5a482e7954aaeeb49f4927b889f68acbb1efbfd2c0ffe96d1210ef600"}
Oct 30 22:47:28.883700 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.883322    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf" event={"ID":"f7fafd0e-2894-4f16-8772-1890eac3ae49","Type":"ContainerDied","Data":"736291494e1ba3963df76f703f188dc1ddffbddd1996486269aa7772feda6e81"}
Oct 30 22:47:28.883700 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.883348    4473 scope.go:122] "RemoveContainer" containerID="2554a5d89d0b0c91bd74de5b8f76243cd99029d592d49e47efe9f1b4b0ee6ea8"
Oct 30 22:47:28.892030 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.892009    4473 scope.go:122] "RemoveContainer" containerID="40ad8bf5a482e7954aaeeb49f4927b889f68acbb1efbfd2c0ffe96d1210ef600"
Oct 30 22:47:28.899473 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.899456    4473 scope.go:122] "RemoveContainer" containerID="4d943f44c70d0861dcd6901b2fefa34dc60328d6a68cbb5591f05c4604363379"
Oct 30 22:47:28.906760 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.906744    4473 scope.go:122] "RemoveContainer" containerID="2554a5d89d0b0c91bd74de5b8f76243cd99029d592d49e47efe9f1b4b0ee6ea8"
Oct 30 22:47:28.907071 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:47:28.907048    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2554a5d89d0b0c91bd74de5b8f76243cd99029d592d49e47efe9f1b4b0ee6ea8\": container with ID starting with 2554a5d89d0b0c91bd74de5b8f76243cd99029d592d49e47efe9f1b4b0ee6ea8 not found: ID does not exist" containerID="2554a5d89d0b0c91bd74de5b8f76243cd99029d592d49e47efe9f1b4b0ee6ea8"
Oct 30 22:47:28.907161 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.907078    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2554a5d89d0b0c91bd74de5b8f76243cd99029d592d49e47efe9f1b4b0ee6ea8"} err="failed to get container status \"2554a5d89d0b0c91bd74de5b8f76243cd99029d592d49e47efe9f1b4b0ee6ea8\": rpc error: code = NotFound desc = could not find container \"2554a5d89d0b0c91bd74de5b8f76243cd99029d592d49e47efe9f1b4b0ee6ea8\": container with ID starting with 2554a5d89d0b0c91bd74de5b8f76243cd99029d592d49e47efe9f1b4b0ee6ea8 not found: ID does not exist"
Oct 30 22:47:28.907161 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.907095    4473 scope.go:122] "RemoveContainer" containerID="40ad8bf5a482e7954aaeeb49f4927b889f68acbb1efbfd2c0ffe96d1210ef600"
Oct 30 22:47:28.907352 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:47:28.907332    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ad8bf5a482e7954aaeeb49f4927b889f68acbb1efbfd2c0ffe96d1210ef600\": container with ID starting with 40ad8bf5a482e7954aaeeb49f4927b889f68acbb1efbfd2c0ffe96d1210ef600 not found: ID does not exist" containerID="40ad8bf5a482e7954aaeeb49f4927b889f68acbb1efbfd2c0ffe96d1210ef600"
Oct 30 22:47:28.907400 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.907357    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ad8bf5a482e7954aaeeb49f4927b889f68acbb1efbfd2c0ffe96d1210ef600"} err="failed to get container status \"40ad8bf5a482e7954aaeeb49f4927b889f68acbb1efbfd2c0ffe96d1210ef600\": rpc error: code = NotFound desc = could not find container \"40ad8bf5a482e7954aaeeb49f4927b889f68acbb1efbfd2c0ffe96d1210ef600\": container with ID starting with 40ad8bf5a482e7954aaeeb49f4927b889f68acbb1efbfd2c0ffe96d1210ef600 not found: ID does not exist"
Oct 30 22:47:28.907400 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.907372    4473 scope.go:122] "RemoveContainer" containerID="4d943f44c70d0861dcd6901b2fefa34dc60328d6a68cbb5591f05c4604363379"
Oct 30 22:47:28.907618 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:47:28.907600    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d943f44c70d0861dcd6901b2fefa34dc60328d6a68cbb5591f05c4604363379\": container with ID starting with 4d943f44c70d0861dcd6901b2fefa34dc60328d6a68cbb5591f05c4604363379 not found: ID does not exist" containerID="4d943f44c70d0861dcd6901b2fefa34dc60328d6a68cbb5591f05c4604363379"
Oct 30 22:47:28.907660 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:28.907624    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d943f44c70d0861dcd6901b2fefa34dc60328d6a68cbb5591f05c4604363379"} err="failed to get container status \"4d943f44c70d0861dcd6901b2fefa34dc60328d6a68cbb5591f05c4604363379\": rpc error: code = NotFound desc = could not find container \"4d943f44c70d0861dcd6901b2fefa34dc60328d6a68cbb5591f05c4604363379\": container with ID starting with 4d943f44c70d0861dcd6901b2fefa34dc60328d6a68cbb5591f05c4604363379 not found: ID does not exist"
Oct 30 22:47:29.133910 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:29.133824    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"]
Oct 30 22:47:29.137937 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:29.137912    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-canary-stop-v2-predictor-7677dd656b-dbpdf"]
Oct 30 22:47:29.886178 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:29.886148    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" path="/var/lib/kubelet/pods/826fa7fe-6eea-476c-9e86-bfbac2c90704/volumes"
Oct 30 22:47:29.886789 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:29.886769    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" path="/var/lib/kubelet/pods/f7fafd0e-2894-4f16-8772-1890eac3ae49/volumes"
Oct 30 22:47:31.524358 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:31.524325    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:47:32.469219 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:32.469188    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:32.800127 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:32.800023    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:33.259302 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.259268    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"]
Oct 30 22:47:33.259596 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.259584    4473 cpu_manager.go:414] "containerMap: removing container" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="storage-initializer"
Oct 30 22:47:33.259640 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.259598    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="storage-initializer"
Oct 30 22:47:33.259640 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.259609    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kserve-container"
Oct 30 22:47:33.259640 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.259614    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kserve-container"
Oct 30 22:47:33.259640 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.259630    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kube-rbac-proxy"
Oct 30 22:47:33.259640 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.259635    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kube-rbac-proxy"
Oct 30 22:47:33.259794 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.259642    4473 cpu_manager.go:414] "containerMap: removing container" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kube-rbac-proxy"
Oct 30 22:47:33.259794 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.259648    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kube-rbac-proxy"
Oct 30 22:47:33.259794 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.259654    4473 cpu_manager.go:414] "containerMap: removing container" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kserve-container"
Oct 30 22:47:33.259794 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.259659    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kserve-container"
Oct 30 22:47:33.259794 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.259678    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="storage-initializer"
Oct 30 22:47:33.259794 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.259682    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="storage-initializer"
Oct 30 22:47:33.259794 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.259728    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kserve-container"
Oct 30 22:47:33.259794 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.259740    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kserve-container"
Oct 30 22:47:33.259794 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.259747    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7fafd0e-2894-4f16-8772-1890eac3ae49" containerName="kube-rbac-proxy"
Oct 30 22:47:33.259794 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.259753    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="826fa7fe-6eea-476c-9e86-bfbac2c90704" containerName="kube-rbac-proxy"
Oct 30 22:47:33.264451 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.264431    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:33.369135 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.369101    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"seaweedfs-s3-creds\""
Oct 30 22:47:33.369135 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.369115    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-lightgbm-predictor-serving-cert\""
Oct 30 22:47:33.369135 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.369130    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"openshift-service-ca.crt\""
Oct 30 22:47:33.369400 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.369170    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:47:33.369400 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.369166    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"kube-root-ca.crt\""
Oct 30 22:47:33.369400 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.369194    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Oct 30 22:47:33.369400 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.369208    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"default-dockercfg-4v4s8\""
Oct 30 22:47:33.370663 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.370646    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"]
Oct 30 22:47:33.434210 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.434174    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a61e24d-5452-468d-b771-26595252b597-cabundle-cert\") pod \"isvc-lightgbm-predictor-856cd7659f-grvnz\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:33.434364 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.434232    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a61e24d-5452-468d-b771-26595252b597-proxy-tls\") pod \"isvc-lightgbm-predictor-856cd7659f-grvnz\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:33.434364 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.434254    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq8hx\" (UniqueName: \"kubernetes.io/projected/6a61e24d-5452-468d-b771-26595252b597-kube-api-access-zq8hx\") pod \"isvc-lightgbm-predictor-856cd7659f-grvnz\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:33.434364 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.434330    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a61e24d-5452-468d-b771-26595252b597-kserve-provision-location\") pod \"isvc-lightgbm-predictor-856cd7659f-grvnz\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:33.434364 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.434357    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a61e24d-5452-468d-b771-26595252b597-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-856cd7659f-grvnz\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:33.535452 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.535367    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a61e24d-5452-468d-b771-26595252b597-proxy-tls\") pod \"isvc-lightgbm-predictor-856cd7659f-grvnz\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:33.535452 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.535403    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq8hx\" (UniqueName: \"kubernetes.io/projected/6a61e24d-5452-468d-b771-26595252b597-kube-api-access-zq8hx\") pod \"isvc-lightgbm-predictor-856cd7659f-grvnz\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:33.535452 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.535429    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a61e24d-5452-468d-b771-26595252b597-kserve-provision-location\") pod \"isvc-lightgbm-predictor-856cd7659f-grvnz\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:33.535722 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.535456    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a61e24d-5452-468d-b771-26595252b597-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-856cd7659f-grvnz\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:33.535722 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.535491    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a61e24d-5452-468d-b771-26595252b597-cabundle-cert\") pod \"isvc-lightgbm-predictor-856cd7659f-grvnz\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:33.535887 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.535858    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a61e24d-5452-468d-b771-26595252b597-kserve-provision-location\") pod \"isvc-lightgbm-predictor-856cd7659f-grvnz\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:33.536189 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.536168    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a61e24d-5452-468d-b771-26595252b597-cabundle-cert\") pod \"isvc-lightgbm-predictor-856cd7659f-grvnz\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:33.536289 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.536251    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a61e24d-5452-468d-b771-26595252b597-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-856cd7659f-grvnz\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:33.538060 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.538038    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a61e24d-5452-468d-b771-26595252b597-proxy-tls\") pod \"isvc-lightgbm-predictor-856cd7659f-grvnz\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:33.594488 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.594457    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq8hx\" (UniqueName: \"kubernetes.io/projected/6a61e24d-5452-468d-b771-26595252b597-kube-api-access-zq8hx\") pod \"isvc-lightgbm-predictor-856cd7659f-grvnz\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:33.881742 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:33.881642    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:34.304559 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:34.304534    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"]
Oct 30 22:47:34.306673 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:47:34.306643    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a61e24d_5452_468d_b771_26595252b597.slice/crio-6cd3dc76e3be11b53a6d3f7821987d95792d1212e673707e3539d4ef87363aff WatchSource:0}: Error finding container 6cd3dc76e3be11b53a6d3f7821987d95792d1212e673707e3539d4ef87363aff: Status 404 returned error can't find the container with id 6cd3dc76e3be11b53a6d3f7821987d95792d1212e673707e3539d4ef87363aff
Oct 30 22:47:34.914078 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:34.914042    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" event={"ID":"6a61e24d-5452-468d-b771-26595252b597","Type":"ContainerStarted","Data":"4e58fa18e44be5ffb6f1ceb1f453a37112d17b0f9fc48e0a4f3f1e88d9aa6b5b"}
Oct 30 22:47:34.914580 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:34.914100    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" event={"ID":"6a61e24d-5452-468d-b771-26595252b597","Type":"ContainerStarted","Data":"6cd3dc76e3be11b53a6d3f7821987d95792d1212e673707e3539d4ef87363aff"}
Oct 30 22:47:38.932918 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:38.932882    4473 generic.go:356] "Generic (PLEG): container finished" podID="6a61e24d-5452-468d-b771-26595252b597" containerID="4e58fa18e44be5ffb6f1ceb1f453a37112d17b0f9fc48e0a4f3f1e88d9aa6b5b" exitCode=0
Oct 30 22:47:38.933389 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:38.932941    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" event={"ID":"6a61e24d-5452-468d-b771-26595252b597","Type":"ContainerDied","Data":"4e58fa18e44be5ffb6f1ceb1f453a37112d17b0f9fc48e0a4f3f1e88d9aa6b5b"}
Oct 30 22:47:38.934381 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:38.934367    4473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:47:40.458493 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:40.458458    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"]
Oct 30 22:47:40.458925 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:40.458713    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kserve-container" containerID="cri-o://f866e84eb8e0383fce36c9a02d9e14ded5676ae513a9cba5d41390318cbae295" gracePeriod=30
Oct 30 22:47:40.458925 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:40.458749    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kube-rbac-proxy" containerID="cri-o://27514699c8e7f45671e97a56f1ffcaa752c85e2ed5814aa95eee81b7986c62dd" gracePeriod=30
Oct 30 22:47:40.464670 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:40.464645    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:40.945243 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:40.945206    4473 generic.go:356] "Generic (PLEG): container finished" podID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerID="27514699c8e7f45671e97a56f1ffcaa752c85e2ed5814aa95eee81b7986c62dd" exitCode=0
Oct 30 22:47:40.945428 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:40.945286    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" event={"ID":"fce83cb9-ddcb-4ce7-abed-7df75e01b841","Type":"ContainerDied","Data":"27514699c8e7f45671e97a56f1ffcaa752c85e2ed5814aa95eee81b7986c62dd"}
Oct 30 22:47:41.307792 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:47:41.307703    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:47:41.520223 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:41.520182    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.69:8643/healthz\": dial tcp 10.134.0.69:8643: connect: connection refused"
Oct 30 22:47:41.524593 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:41.524561    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Oct 30 22:47:42.808427 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:42.808347    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:43.042898 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.042860    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:43.197526 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.197491    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"]
Oct 30 22:47:43.202687 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.202663    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"
Oct 30 22:47:43.255607 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.255564    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Oct 30 22:47:43.306165 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.306133    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Oct 30 22:47:43.307937 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.307904    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"]
Oct 30 22:47:43.322561 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.322518    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/451105d6-0f02-4705-a44f-0eb10c3dc69b-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p\" (UID: \"451105d6-0f02-4705-a44f-0eb10c3dc69b\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"
Oct 30 22:47:43.322699 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.322570    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv9dz\" (UniqueName: \"kubernetes.io/projected/451105d6-0f02-4705-a44f-0eb10c3dc69b-kube-api-access-gv9dz\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p\" (UID: \"451105d6-0f02-4705-a44f-0eb10c3dc69b\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"
Oct 30 22:47:43.322747 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.322688    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/451105d6-0f02-4705-a44f-0eb10c3dc69b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p\" (UID: \"451105d6-0f02-4705-a44f-0eb10c3dc69b\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"
Oct 30 22:47:43.322786 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.322744    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/451105d6-0f02-4705-a44f-0eb10c3dc69b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p\" (UID: \"451105d6-0f02-4705-a44f-0eb10c3dc69b\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"
Oct 30 22:47:43.423828 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.423792    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv9dz\" (UniqueName: \"kubernetes.io/projected/451105d6-0f02-4705-a44f-0eb10c3dc69b-kube-api-access-gv9dz\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p\" (UID: \"451105d6-0f02-4705-a44f-0eb10c3dc69b\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"
Oct 30 22:47:43.424048 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.423848    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/451105d6-0f02-4705-a44f-0eb10c3dc69b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p\" (UID: \"451105d6-0f02-4705-a44f-0eb10c3dc69b\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"
Oct 30 22:47:43.424048 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.423867    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/451105d6-0f02-4705-a44f-0eb10c3dc69b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p\" (UID: \"451105d6-0f02-4705-a44f-0eb10c3dc69b\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"
Oct 30 22:47:43.424048 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.423942    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/451105d6-0f02-4705-a44f-0eb10c3dc69b-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p\" (UID: \"451105d6-0f02-4705-a44f-0eb10c3dc69b\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"
Oct 30 22:47:43.424357 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.424330    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/451105d6-0f02-4705-a44f-0eb10c3dc69b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p\" (UID: \"451105d6-0f02-4705-a44f-0eb10c3dc69b\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"
Oct 30 22:47:43.424600 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.424578    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/451105d6-0f02-4705-a44f-0eb10c3dc69b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p\" (UID: \"451105d6-0f02-4705-a44f-0eb10c3dc69b\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"
Oct 30 22:47:43.426761 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.426738    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/451105d6-0f02-4705-a44f-0eb10c3dc69b-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p\" (UID: \"451105d6-0f02-4705-a44f-0eb10c3dc69b\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"
Oct 30 22:47:43.482036 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.481939    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv9dz\" (UniqueName: \"kubernetes.io/projected/451105d6-0f02-4705-a44f-0eb10c3dc69b-kube-api-access-gv9dz\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p\" (UID: \"451105d6-0f02-4705-a44f-0eb10c3dc69b\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"
Oct 30 22:47:43.522874 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.522801    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"
Oct 30 22:47:43.966749 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:43.966653    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"]
Oct 30 22:47:43.971474 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:47:43.971441    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451105d6_0f02_4705_a44f_0eb10c3dc69b.slice/crio-5dbf40a77ecd0d362a883eea8d3cf40d1d34dde60f024258afd88945b65f3025 WatchSource:0}: Error finding container 5dbf40a77ecd0d362a883eea8d3cf40d1d34dde60f024258afd88945b65f3025: Status 404 returned error can't find the container with id 5dbf40a77ecd0d362a883eea8d3cf40d1d34dde60f024258afd88945b65f3025
Oct 30 22:47:44.976569 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:44.976529    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p" event={"ID":"451105d6-0f02-4705-a44f-0eb10c3dc69b","Type":"ContainerStarted","Data":"196a54ee510c1ec1c0ed27a8993b7bfed58a476c44e4f490a4323ffe19cd6d47"}
Oct 30 22:47:44.977064 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:44.976602    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p" event={"ID":"451105d6-0f02-4705-a44f-0eb10c3dc69b","Type":"ContainerStarted","Data":"5dbf40a77ecd0d362a883eea8d3cf40d1d34dde60f024258afd88945b65f3025"}
Oct 30 22:47:46.233353 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.233293    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:47:46.250113 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.250083    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fce83cb9-ddcb-4ce7-abed-7df75e01b841-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fce83cb9-ddcb-4ce7-abed-7df75e01b841-cabundle-cert\") pod \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") "
Oct 30 22:47:46.250250 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.250133    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kube-rbac-proxy-sar-config\") pod \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") "
Oct 30 22:47:46.250250 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.250176    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fce83cb9-ddcb-4ce7-abed-7df75e01b841-proxy-tls\" (UniqueName: \"kubernetes.io/secret/fce83cb9-ddcb-4ce7-abed-7df75e01b841-proxy-tls\") pod \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") "
Oct 30 22:47:46.250250 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.250208    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kube-api-access-9vpvg\" (UniqueName: \"kubernetes.io/projected/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kube-api-access-9vpvg\") pod \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") "
Oct 30 22:47:46.250250 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.250242    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kserve-provision-location\") pod \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\" (UID: \"fce83cb9-ddcb-4ce7-abed-7df75e01b841\") "
Oct 30 22:47:46.250754 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.250727    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kserve-provision-location" pod "fce83cb9-ddcb-4ce7-abed-7df75e01b841" (UID: "fce83cb9-ddcb-4ce7-abed-7df75e01b841"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:46.250754 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.250745    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kube-rbac-proxy-sar-config" pod "fce83cb9-ddcb-4ce7-abed-7df75e01b841" (UID: "fce83cb9-ddcb-4ce7-abed-7df75e01b841"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:46.251052 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.251022    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce83cb9-ddcb-4ce7-abed-7df75e01b841-cabundle-cert" pod "fce83cb9-ddcb-4ce7-abed-7df75e01b841" (UID: "fce83cb9-ddcb-4ce7-abed-7df75e01b841"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:46.253215 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.253195    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fce83cb9-ddcb-4ce7-abed-7df75e01b841-proxy-tls" pod "fce83cb9-ddcb-4ce7-abed-7df75e01b841" (UID: "fce83cb9-ddcb-4ce7-abed-7df75e01b841"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:46.253305 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.253261    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kube-api-access-9vpvg" pod "fce83cb9-ddcb-4ce7-abed-7df75e01b841" (UID: "fce83cb9-ddcb-4ce7-abed-7df75e01b841"). InnerVolumeSpecName "kube-api-access-9vpvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:46.351260 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.351229    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fce83cb9-ddcb-4ce7-abed-7df75e01b841-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.351260 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.351257    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.351260 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.351268    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fce83cb9-ddcb-4ce7-abed-7df75e01b841-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.351548 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.351278    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vpvg\" (UniqueName: \"kubernetes.io/projected/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kube-api-access-9vpvg\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.351548 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.351289    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce83cb9-ddcb-4ce7-abed-7df75e01b841-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.987465 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.987373    4473 generic.go:356] "Generic (PLEG): container finished" podID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerID="f866e84eb8e0383fce36c9a02d9e14ded5676ae513a9cba5d41390318cbae295" exitCode=0
Oct 30 22:47:46.987465 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.987442    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"
Oct 30 22:47:46.987685 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.987460    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" event={"ID":"fce83cb9-ddcb-4ce7-abed-7df75e01b841","Type":"ContainerDied","Data":"f866e84eb8e0383fce36c9a02d9e14ded5676ae513a9cba5d41390318cbae295"}
Oct 30 22:47:46.987685 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.987538    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r" event={"ID":"fce83cb9-ddcb-4ce7-abed-7df75e01b841","Type":"ContainerDied","Data":"c2ffe723298abe23214e1ffc0644451815fed75c6f5077bc7e7d2d89698826ab"}
Oct 30 22:47:46.987685 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.987583    4473 scope.go:122] "RemoveContainer" containerID="27514699c8e7f45671e97a56f1ffcaa752c85e2ed5814aa95eee81b7986c62dd"
Oct 30 22:47:46.989782 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.989755    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" event={"ID":"6a61e24d-5452-468d-b771-26595252b597","Type":"ContainerStarted","Data":"062ab780807392e803119be77e1bf481cbf32fb89436e3aa67cb35d667681162"}
Oct 30 22:47:46.989899 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.989820    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" event={"ID":"6a61e24d-5452-468d-b771-26595252b597","Type":"ContainerStarted","Data":"ee2ba74b1c8549ba8e9e59d2c5fbee179f0835d15c41e77200cc75d38e703bc9"}
Oct 30 22:47:46.990191 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.990167    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:46.990335 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.990216    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:46.991297 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.991275    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Oct 30 22:47:46.996867 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:46.996851    4473 scope.go:122] "RemoveContainer" containerID="f866e84eb8e0383fce36c9a02d9e14ded5676ae513a9cba5d41390318cbae295"
Oct 30 22:47:47.008985 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:47.008966    4473 scope.go:122] "RemoveContainer" containerID="20571980f3837a8170b0467d32c7bb65c9822b6f045253fa526c8c62c35d090b"
Oct 30 22:47:47.016226 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:47.016209    4473 scope.go:122] "RemoveContainer" containerID="27514699c8e7f45671e97a56f1ffcaa752c85e2ed5814aa95eee81b7986c62dd"
Oct 30 22:47:47.016503 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:47:47.016485    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27514699c8e7f45671e97a56f1ffcaa752c85e2ed5814aa95eee81b7986c62dd\": container with ID starting with 27514699c8e7f45671e97a56f1ffcaa752c85e2ed5814aa95eee81b7986c62dd not found: ID does not exist" containerID="27514699c8e7f45671e97a56f1ffcaa752c85e2ed5814aa95eee81b7986c62dd"
Oct 30 22:47:47.016554 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:47.016513    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27514699c8e7f45671e97a56f1ffcaa752c85e2ed5814aa95eee81b7986c62dd"} err="failed to get container status \"27514699c8e7f45671e97a56f1ffcaa752c85e2ed5814aa95eee81b7986c62dd\": rpc error: code = NotFound desc = could not find container \"27514699c8e7f45671e97a56f1ffcaa752c85e2ed5814aa95eee81b7986c62dd\": container with ID starting with 27514699c8e7f45671e97a56f1ffcaa752c85e2ed5814aa95eee81b7986c62dd not found: ID does not exist"
Oct 30 22:47:47.016554 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:47.016532    4473 scope.go:122] "RemoveContainer" containerID="f866e84eb8e0383fce36c9a02d9e14ded5676ae513a9cba5d41390318cbae295"
Oct 30 22:47:47.016770 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:47:47.016752    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f866e84eb8e0383fce36c9a02d9e14ded5676ae513a9cba5d41390318cbae295\": container with ID starting with f866e84eb8e0383fce36c9a02d9e14ded5676ae513a9cba5d41390318cbae295 not found: ID does not exist" containerID="f866e84eb8e0383fce36c9a02d9e14ded5676ae513a9cba5d41390318cbae295"
Oct 30 22:47:47.016814 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:47.016778    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f866e84eb8e0383fce36c9a02d9e14ded5676ae513a9cba5d41390318cbae295"} err="failed to get container status \"f866e84eb8e0383fce36c9a02d9e14ded5676ae513a9cba5d41390318cbae295\": rpc error: code = NotFound desc = could not find container \"f866e84eb8e0383fce36c9a02d9e14ded5676ae513a9cba5d41390318cbae295\": container with ID starting with f866e84eb8e0383fce36c9a02d9e14ded5676ae513a9cba5d41390318cbae295 not found: ID does not exist"
Oct 30 22:47:47.016814 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:47.016794    4473 scope.go:122] "RemoveContainer" containerID="20571980f3837a8170b0467d32c7bb65c9822b6f045253fa526c8c62c35d090b"
Oct 30 22:47:47.017041 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:47:47.017025    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20571980f3837a8170b0467d32c7bb65c9822b6f045253fa526c8c62c35d090b\": container with ID starting with 20571980f3837a8170b0467d32c7bb65c9822b6f045253fa526c8c62c35d090b not found: ID does not exist" containerID="20571980f3837a8170b0467d32c7bb65c9822b6f045253fa526c8c62c35d090b"
Oct 30 22:47:47.017088 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:47.017048    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20571980f3837a8170b0467d32c7bb65c9822b6f045253fa526c8c62c35d090b"} err="failed to get container status \"20571980f3837a8170b0467d32c7bb65c9822b6f045253fa526c8c62c35d090b\": rpc error: code = NotFound desc = could not find container \"20571980f3837a8170b0467d32c7bb65c9822b6f045253fa526c8c62c35d090b\": container with ID starting with 20571980f3837a8170b0467d32c7bb65c9822b6f045253fa526c8c62c35d090b not found: ID does not exist"
Oct 30 22:47:47.169543 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:47.169482    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" podStartSLOduration=2.409421342 podStartE2EDuration="14.169464794s" totalImagesPullingTime="7.725187902s" totalInitContainerRuntime="4.03485555s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:47.16687552 +0000 UTC m=+3285.403084865" watchObservedRunningTime="2026-10-30 22:47:47.169464794 +0000 UTC m=+3285.405674205"
Oct 30 22:47:47.285812 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:47.285720    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"]
Oct 30 22:47:47.294127 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:47.294099    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-pass-predictor-6f9b95b9b4-9xb2r"]
Oct 30 22:47:47.886049 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:47.886021    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" path="/var/lib/kubelet/pods/fce83cb9-ddcb-4ce7-abed-7df75e01b841/volumes"
Oct 30 22:47:48.000879 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:48.000835    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Oct 30 22:47:50.009789 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:50.009756    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p_451105d6-0f02-4705-a44f-0eb10c3dc69b/storage-initializer/0.log"
Oct 30 22:47:50.010209 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:50.009796    4473 generic.go:356] "Generic (PLEG): container finished" podID="451105d6-0f02-4705-a44f-0eb10c3dc69b" containerID="196a54ee510c1ec1c0ed27a8993b7bfed58a476c44e4f490a4323ffe19cd6d47" exitCode=1
Oct 30 22:47:50.010209 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:50.009872    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p" event={"ID":"451105d6-0f02-4705-a44f-0eb10c3dc69b","Type":"ContainerDied","Data":"196a54ee510c1ec1c0ed27a8993b7bfed58a476c44e4f490a4323ffe19cd6d47"}
Oct 30 22:47:51.016184 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:51.016156    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p_451105d6-0f02-4705-a44f-0eb10c3dc69b/storage-initializer/0.log"
Oct 30 22:47:51.016585 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:51.016218    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p" event={"ID":"451105d6-0f02-4705-a44f-0eb10c3dc69b","Type":"ContainerStarted","Data":"d135527ea15c998dfe59671fa73a9b39e7b943fc1561a81e69c0fde0f7f69c07"}
Oct 30 22:47:52.992929 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:52.992889    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"]
Oct 30 22:47:52.993373 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:52.993124    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p" podUID="451105d6-0f02-4705-a44f-0eb10c3dc69b" containerName="storage-initializer" containerID="cri-o://d135527ea15c998dfe59671fa73a9b39e7b943fc1561a81e69c0fde0f7f69c07" gracePeriod=30
Oct 30 22:47:53.005222 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:53.005197    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:47:53.005682 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:53.005656    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Oct 30 22:47:53.021461 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:53.021440    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:54.835070 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:54.835039    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p_451105d6-0f02-4705-a44f-0eb10c3dc69b/storage-initializer/1.log"
Oct 30 22:47:54.835431 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:54.835417    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p_451105d6-0f02-4705-a44f-0eb10c3dc69b/storage-initializer/0.log"
Oct 30 22:47:54.837735 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:54.837722    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"
Oct 30 22:47:54.910563 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:54.910536    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/451105d6-0f02-4705-a44f-0eb10c3dc69b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/451105d6-0f02-4705-a44f-0eb10c3dc69b-kube-rbac-proxy-sar-config\") pod \"451105d6-0f02-4705-a44f-0eb10c3dc69b\" (UID: \"451105d6-0f02-4705-a44f-0eb10c3dc69b\") "
Oct 30 22:47:54.910731 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:54.910575    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/451105d6-0f02-4705-a44f-0eb10c3dc69b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/451105d6-0f02-4705-a44f-0eb10c3dc69b-proxy-tls\") pod \"451105d6-0f02-4705-a44f-0eb10c3dc69b\" (UID: \"451105d6-0f02-4705-a44f-0eb10c3dc69b\") "
Oct 30 22:47:54.910731 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:54.910631    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451105d6-0f02-4705-a44f-0eb10c3dc69b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/451105d6-0f02-4705-a44f-0eb10c3dc69b-kserve-provision-location\") pod \"451105d6-0f02-4705-a44f-0eb10c3dc69b\" (UID: \"451105d6-0f02-4705-a44f-0eb10c3dc69b\") "
Oct 30 22:47:54.910731 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:54.910676    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/451105d6-0f02-4705-a44f-0eb10c3dc69b-kube-api-access-gv9dz\" (UniqueName: \"kubernetes.io/projected/451105d6-0f02-4705-a44f-0eb10c3dc69b-kube-api-access-gv9dz\") pod \"451105d6-0f02-4705-a44f-0eb10c3dc69b\" (UID: \"451105d6-0f02-4705-a44f-0eb10c3dc69b\") "
Oct 30 22:47:54.910920 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:54.910898    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/451105d6-0f02-4705-a44f-0eb10c3dc69b-kube-rbac-proxy-sar-config" pod "451105d6-0f02-4705-a44f-0eb10c3dc69b" (UID: "451105d6-0f02-4705-a44f-0eb10c3dc69b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:54.910987 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:54.910925    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451105d6-0f02-4705-a44f-0eb10c3dc69b-kserve-provision-location" pod "451105d6-0f02-4705-a44f-0eb10c3dc69b" (UID: "451105d6-0f02-4705-a44f-0eb10c3dc69b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:54.912913 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:54.912885    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/451105d6-0f02-4705-a44f-0eb10c3dc69b-proxy-tls" pod "451105d6-0f02-4705-a44f-0eb10c3dc69b" (UID: "451105d6-0f02-4705-a44f-0eb10c3dc69b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:54.913032 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:54.912971    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451105d6-0f02-4705-a44f-0eb10c3dc69b-kube-api-access-gv9dz" pod "451105d6-0f02-4705-a44f-0eb10c3dc69b" (UID: "451105d6-0f02-4705-a44f-0eb10c3dc69b"). InnerVolumeSpecName "kube-api-access-gv9dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:55.011260 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.011226    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/451105d6-0f02-4705-a44f-0eb10c3dc69b-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:55.011260 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.011260    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv9dz\" (UniqueName: \"kubernetes.io/projected/451105d6-0f02-4705-a44f-0eb10c3dc69b-kube-api-access-gv9dz\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:55.011260 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.011276    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/451105d6-0f02-4705-a44f-0eb10c3dc69b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:55.011522 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.011286    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/451105d6-0f02-4705-a44f-0eb10c3dc69b-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:55.033726 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.033698    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p_451105d6-0f02-4705-a44f-0eb10c3dc69b/storage-initializer/1.log"
Oct 30 22:47:55.034099 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.034080    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7476d02544_isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p_451105d6-0f02-4705-a44f-0eb10c3dc69b/storage-initializer/0.log"
Oct 30 22:47:55.034229 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.034121    4473 generic.go:356] "Generic (PLEG): container finished" podID="451105d6-0f02-4705-a44f-0eb10c3dc69b" containerID="d135527ea15c998dfe59671fa73a9b39e7b943fc1561a81e69c0fde0f7f69c07" exitCode=1
Oct 30 22:47:55.034229 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.034155    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p" event={"ID":"451105d6-0f02-4705-a44f-0eb10c3dc69b","Type":"ContainerDied","Data":"d135527ea15c998dfe59671fa73a9b39e7b943fc1561a81e69c0fde0f7f69c07"}
Oct 30 22:47:55.034229 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.034189    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"
Oct 30 22:47:55.034391 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.034253    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p" event={"ID":"451105d6-0f02-4705-a44f-0eb10c3dc69b","Type":"ContainerDied","Data":"5dbf40a77ecd0d362a883eea8d3cf40d1d34dde60f024258afd88945b65f3025"}
Oct 30 22:47:55.034391 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.034294    4473 scope.go:122] "RemoveContainer" containerID="d135527ea15c998dfe59671fa73a9b39e7b943fc1561a81e69c0fde0f7f69c07"
Oct 30 22:47:55.043149 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.043133    4473 scope.go:122] "RemoveContainer" containerID="196a54ee510c1ec1c0ed27a8993b7bfed58a476c44e4f490a4323ffe19cd6d47"
Oct 30 22:47:55.050676 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.050660    4473 scope.go:122] "RemoveContainer" containerID="d135527ea15c998dfe59671fa73a9b39e7b943fc1561a81e69c0fde0f7f69c07"
Oct 30 22:47:55.050937 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:47:55.050918    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d135527ea15c998dfe59671fa73a9b39e7b943fc1561a81e69c0fde0f7f69c07\": container with ID starting with d135527ea15c998dfe59671fa73a9b39e7b943fc1561a81e69c0fde0f7f69c07 not found: ID does not exist" containerID="d135527ea15c998dfe59671fa73a9b39e7b943fc1561a81e69c0fde0f7f69c07"
Oct 30 22:47:55.051020 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.050964    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d135527ea15c998dfe59671fa73a9b39e7b943fc1561a81e69c0fde0f7f69c07"} err="failed to get container status \"d135527ea15c998dfe59671fa73a9b39e7b943fc1561a81e69c0fde0f7f69c07\": rpc error: code = NotFound desc = could not find container \"d135527ea15c998dfe59671fa73a9b39e7b943fc1561a81e69c0fde0f7f69c07\": container with ID starting with d135527ea15c998dfe59671fa73a9b39e7b943fc1561a81e69c0fde0f7f69c07 not found: ID does not exist"
Oct 30 22:47:55.051020 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.050991    4473 scope.go:122] "RemoveContainer" containerID="196a54ee510c1ec1c0ed27a8993b7bfed58a476c44e4f490a4323ffe19cd6d47"
Oct 30 22:47:55.051219 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:47:55.051203    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196a54ee510c1ec1c0ed27a8993b7bfed58a476c44e4f490a4323ffe19cd6d47\": container with ID starting with 196a54ee510c1ec1c0ed27a8993b7bfed58a476c44e4f490a4323ffe19cd6d47 not found: ID does not exist" containerID="196a54ee510c1ec1c0ed27a8993b7bfed58a476c44e4f490a4323ffe19cd6d47"
Oct 30 22:47:55.051256 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.051225    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196a54ee510c1ec1c0ed27a8993b7bfed58a476c44e4f490a4323ffe19cd6d47"} err="failed to get container status \"196a54ee510c1ec1c0ed27a8993b7bfed58a476c44e4f490a4323ffe19cd6d47\": rpc error: code = NotFound desc = could not find container \"196a54ee510c1ec1c0ed27a8993b7bfed58a476c44e4f490a4323ffe19cd6d47\": container with ID starting with 196a54ee510c1ec1c0ed27a8993b7bfed58a476c44e4f490a4323ffe19cd6d47 not found: ID does not exist"
Oct 30 22:47:55.267017 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.266924    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"]
Oct 30 22:47:55.270676 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.270652    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-custom-fail-predictor-8488f874dd-zv56p"]
Oct 30 22:47:55.886245 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:55.886214    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="451105d6-0f02-4705-a44f-0eb10c3dc69b" path="/var/lib/kubelet/pods/451105d6-0f02-4705-a44f-0eb10c3dc69b/volumes"
Oct 30 22:47:57.493276 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.493242    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:57.677049 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.677018    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:57.886383 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.886304    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"]
Oct 30 22:47:57.886610 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.886598    4473 cpu_manager.go:414] "containerMap: removing container" podUID="451105d6-0f02-4705-a44f-0eb10c3dc69b" containerName="storage-initializer"
Oct 30 22:47:57.886657 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.886612    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="451105d6-0f02-4705-a44f-0eb10c3dc69b" containerName="storage-initializer"
Oct 30 22:47:57.886657 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.886633    4473 cpu_manager.go:414] "containerMap: removing container" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="storage-initializer"
Oct 30 22:47:57.886657 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.886638    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="storage-initializer"
Oct 30 22:47:57.886657 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.886645    4473 cpu_manager.go:414] "containerMap: removing container" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kube-rbac-proxy"
Oct 30 22:47:57.886657 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.886650    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kube-rbac-proxy"
Oct 30 22:47:57.886834 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.886661    4473 cpu_manager.go:414] "containerMap: removing container" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kserve-container"
Oct 30 22:47:57.886834 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.886666    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kserve-container"
Oct 30 22:47:57.886834 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.886677    4473 cpu_manager.go:414] "containerMap: removing container" podUID="451105d6-0f02-4705-a44f-0eb10c3dc69b" containerName="storage-initializer"
Oct 30 22:47:57.886834 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.886681    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="451105d6-0f02-4705-a44f-0eb10c3dc69b" containerName="storage-initializer"
Oct 30 22:47:57.886834 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.886737    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="451105d6-0f02-4705-a44f-0eb10c3dc69b" containerName="storage-initializer"
Oct 30 22:47:57.886834 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.886748    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kube-rbac-proxy"
Oct 30 22:47:57.886834 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.886755    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce83cb9-ddcb-4ce7-abed-7df75e01b841" containerName="kserve-container"
Oct 30 22:47:57.887089 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.886854    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="451105d6-0f02-4705-a44f-0eb10c3dc69b" containerName="storage-initializer"
Oct 30 22:47:57.891235 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.891214    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:47:57.933702 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.933675    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szx48\" (UniqueName: \"kubernetes.io/projected/4c6d9f02-368c-4f5e-88fa-789788c8677c-kube-api-access-szx48\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:47:57.933811 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.933709    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c6d9f02-368c-4f5e-88fa-789788c8677c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:47:57.933811 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.933787    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c6d9f02-368c-4f5e-88fa-789788c8677c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:47:57.933889 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.933828    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c6d9f02-368c-4f5e-88fa-789788c8677c-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:47:57.933970 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.933930    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c6d9f02-368c-4f5e-88fa-789788c8677c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:47:57.995377 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.995274    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"openshift-service-ca.crt\""
Oct 30 22:47:57.995377 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.995294    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Oct 30 22:47:57.995377 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.995324    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:47:57.995377 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.995333    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Oct 30 22:47:57.995377 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.995330    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"storage-config\""
Oct 30 22:47:57.995377 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.995383    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7476d02544\"/\"kube-root-ca.crt\""
Oct 30 22:47:57.995831 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.995435    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7476d02544\"/\"default-dockercfg-cxfkn\""
Oct 30 22:47:57.996923 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:57.996900    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"]
Oct 30 22:47:58.034663 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:58.034635    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c6d9f02-368c-4f5e-88fa-789788c8677c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:47:58.034821 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:58.034681    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szx48\" (UniqueName: \"kubernetes.io/projected/4c6d9f02-368c-4f5e-88fa-789788c8677c-kube-api-access-szx48\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:47:58.034821 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:58.034711    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c6d9f02-368c-4f5e-88fa-789788c8677c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:47:58.034821 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:58.034751    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c6d9f02-368c-4f5e-88fa-789788c8677c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:47:58.034821 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:58.034780    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c6d9f02-368c-4f5e-88fa-789788c8677c-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:47:58.035163 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:58.035137    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c6d9f02-368c-4f5e-88fa-789788c8677c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:47:58.035501 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:58.035479    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c6d9f02-368c-4f5e-88fa-789788c8677c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:47:58.035501 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:58.035513    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c6d9f02-368c-4f5e-88fa-789788c8677c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:47:58.037303 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:58.037284    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c6d9f02-368c-4f5e-88fa-789788c8677c-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:47:58.093549 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:58.093511    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szx48\" (UniqueName: \"kubernetes.io/projected/4c6d9f02-368c-4f5e-88fa-789788c8677c-kube-api-access-szx48\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") " pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:47:58.208436 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:58.208395    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:47:58.633176 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:58.633152    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"]
Oct 30 22:47:58.635470 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:47:58.635443    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6d9f02_368c_4f5e_88fa_789788c8677c.slice/crio-954a3a688f15b6b04a7158c353122bfb599dce184685dc1eb74ebacbc5d59f73 WatchSource:0}: Error finding container 954a3a688f15b6b04a7158c353122bfb599dce184685dc1eb74ebacbc5d59f73: Status 404 returned error can't find the container with id 954a3a688f15b6b04a7158c353122bfb599dce184685dc1eb74ebacbc5d59f73
Oct 30 22:47:59.053622 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:59.053587    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" event={"ID":"4c6d9f02-368c-4f5e-88fa-789788c8677c","Type":"ContainerStarted","Data":"1cc308c327b074bbfdbb2707d100eeb4e818c3793b5b23a70e6eba255865d6d0"}
Oct 30 22:47:59.053771 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:47:59.053649    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" event={"ID":"4c6d9f02-368c-4f5e-88fa-789788c8677c","Type":"ContainerStarted","Data":"954a3a688f15b6b04a7158c353122bfb599dce184685dc1eb74ebacbc5d59f73"}
Oct 30 22:48:00.065367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:00.065285    4473 generic.go:356] "Generic (PLEG): container finished" podID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerID="1cc308c327b074bbfdbb2707d100eeb4e818c3793b5b23a70e6eba255865d6d0" exitCode=0
Oct 30 22:48:00.065741 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:00.065373    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" event={"ID":"4c6d9f02-368c-4f5e-88fa-789788c8677c","Type":"ContainerDied","Data":"1cc308c327b074bbfdbb2707d100eeb4e818c3793b5b23a70e6eba255865d6d0"}
Oct 30 22:48:01.072248 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:01.072213    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" event={"ID":"4c6d9f02-368c-4f5e-88fa-789788c8677c","Type":"ContainerStarted","Data":"e86d060b4b096721c2b29eacdddd4b7912e3916c3eb712fcbf35117c5ffef14b"}
Oct 30 22:48:01.072664 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:01.072268    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" event={"ID":"4c6d9f02-368c-4f5e-88fa-789788c8677c","Type":"ContainerStarted","Data":"3c540caef259a470c2663d04b4da76420f8853cae69286e0c662fc314cf9509d"}
Oct 30 22:48:01.072664 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:01.072420    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:48:01.244678 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:01.244622    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" podStartSLOduration=3.222185893 podStartE2EDuration="4.244607842s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.022421949s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:01.242168293 +0000 UTC m=+3299.478377639" watchObservedRunningTime="2026-10-30 22:48:01.244607842 +0000 UTC m=+3299.480817187"
Oct 30 22:48:02.076542 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:02.076510    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:48:02.078070 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:02.078041    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Oct 30 22:48:02.191375 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:02.191272    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:48:02.223775 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:02.198175    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:48:03.005702 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:03.005669    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Oct 30 22:48:03.081154 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:03.081119    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Oct 30 22:48:08.086476 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:08.086440    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:48:08.087098 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:08.087068    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Oct 30 22:48:09.876058 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:09.876025    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:11.304003 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:48:11.303967    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:48:13.005895 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:13.005862    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Oct 30 22:48:18.087266 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:18.087231    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Oct 30 22:48:23.005653 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:23.005614    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Oct 30 22:48:28.087074 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:28.087031    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Oct 30 22:48:33.005990 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:33.005929    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Oct 30 22:48:38.087921 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:38.087882    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Oct 30 22:48:41.303852 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:48:41.303816    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:48:43.006421 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:43.006374    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Oct 30 22:48:48.087033 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:48.086990    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Oct 30 22:48:53.006438 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:53.006399    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Oct 30 22:48:58.087567 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:48:58.087526    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Oct 30 22:49:00.881911 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:00.881874    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Oct 30 22:49:08.087080 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:08.087040    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Oct 30 22:49:10.270483 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:10.270436    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:10.882293 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:10.882251    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Oct 30 22:49:11.303827 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:49:11.303791    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:49:18.088161 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:18.088082    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:49:20.882124 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:20.882092    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:49:27.801499 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:27.801464    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"]
Oct 30 22:49:27.801941 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:27.801693    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kserve-container" containerID="cri-o://3c540caef259a470c2663d04b4da76420f8853cae69286e0c662fc314cf9509d" gracePeriod=30
Oct 30 22:49:27.801941 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:27.801732    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kube-rbac-proxy" containerID="cri-o://e86d060b4b096721c2b29eacdddd4b7912e3916c3eb712fcbf35117c5ffef14b" gracePeriod=30
Oct 30 22:49:27.833461 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:27.833434    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:28.081568 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:28.081486    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.72:8643/healthz\": dial tcp 10.134.0.72:8643: connect: connection refused"
Oct 30 22:49:28.087438 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:28.087415    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Oct 30 22:49:28.438199 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:28.438169    4473 generic.go:356] "Generic (PLEG): container finished" podID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerID="e86d060b4b096721c2b29eacdddd4b7912e3916c3eb712fcbf35117c5ffef14b" exitCode=0
Oct 30 22:49:28.438374 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:28.438206    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" event={"ID":"4c6d9f02-368c-4f5e-88fa-789788c8677c","Type":"ContainerDied","Data":"e86d060b4b096721c2b29eacdddd4b7912e3916c3eb712fcbf35117c5ffef14b"}
Oct 30 22:49:30.106824 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:30.106790    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:30.351056 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:30.351023    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:33.082191 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:33.082145    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.72:8643/healthz\": dial tcp 10.134.0.72:8643: connect: connection refused"
Oct 30 22:49:34.908890 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:34.908853    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:35.208684 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.208605    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"]
Oct 30 22:49:35.208889 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.208826    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kserve-container" containerID="cri-o://ee2ba74b1c8549ba8e9e59d2c5fbee179f0835d15c41e77200cc75d38e703bc9" gracePeriod=30
Oct 30 22:49:35.208964 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.208889    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kube-rbac-proxy" containerID="cri-o://062ab780807392e803119be77e1bf481cbf32fb89436e3aa67cb35d667681162" gracePeriod=30
Oct 30 22:49:35.215840 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.215809    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:35.470763 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.470665    4473 generic.go:356] "Generic (PLEG): container finished" podID="6a61e24d-5452-468d-b771-26595252b597" containerID="062ab780807392e803119be77e1bf481cbf32fb89436e3aa67cb35d667681162" exitCode=0
Oct 30 22:49:35.470917 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.470753    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" event={"ID":"6a61e24d-5452-468d-b771-26595252b597","Type":"ContainerDied","Data":"062ab780807392e803119be77e1bf481cbf32fb89436e3aa67cb35d667681162"}
Oct 30 22:49:35.751475 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.751380    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:35.834933 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.834905    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:49:35.935199 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.935163    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6d9f02-368c-4f5e-88fa-789788c8677c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c6d9f02-368c-4f5e-88fa-789788c8677c-kserve-provision-location\") pod \"4c6d9f02-368c-4f5e-88fa-789788c8677c\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") "
Oct 30 22:49:35.935655 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.935229    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c6d9f02-368c-4f5e-88fa-789788c8677c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c6d9f02-368c-4f5e-88fa-789788c8677c-kube-rbac-proxy-sar-config\") pod \"4c6d9f02-368c-4f5e-88fa-789788c8677c\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") "
Oct 30 22:49:35.935655 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.935258    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6d9f02-368c-4f5e-88fa-789788c8677c-kube-api-access-szx48\" (UniqueName: \"kubernetes.io/projected/4c6d9f02-368c-4f5e-88fa-789788c8677c-kube-api-access-szx48\") pod \"4c6d9f02-368c-4f5e-88fa-789788c8677c\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") "
Oct 30 22:49:35.935655 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.935275    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c6d9f02-368c-4f5e-88fa-789788c8677c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c6d9f02-368c-4f5e-88fa-789788c8677c-cabundle-cert\") pod \"4c6d9f02-368c-4f5e-88fa-789788c8677c\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") "
Oct 30 22:49:35.935655 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.935316    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c6d9f02-368c-4f5e-88fa-789788c8677c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c6d9f02-368c-4f5e-88fa-789788c8677c-proxy-tls\") pod \"4c6d9f02-368c-4f5e-88fa-789788c8677c\" (UID: \"4c6d9f02-368c-4f5e-88fa-789788c8677c\") "
Oct 30 22:49:35.935655 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.935504    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6d9f02-368c-4f5e-88fa-789788c8677c-kserve-provision-location" pod "4c6d9f02-368c-4f5e-88fa-789788c8677c" (UID: "4c6d9f02-368c-4f5e-88fa-789788c8677c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:35.935942 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.935650    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6d9f02-368c-4f5e-88fa-789788c8677c-kube-rbac-proxy-sar-config" pod "4c6d9f02-368c-4f5e-88fa-789788c8677c" (UID: "4c6d9f02-368c-4f5e-88fa-789788c8677c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:35.935942 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.935676    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6d9f02-368c-4f5e-88fa-789788c8677c-cabundle-cert" pod "4c6d9f02-368c-4f5e-88fa-789788c8677c" (UID: "4c6d9f02-368c-4f5e-88fa-789788c8677c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:35.935942 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.935908    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c6d9f02-368c-4f5e-88fa-789788c8677c-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:35.936145 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.935983    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c6d9f02-368c-4f5e-88fa-789788c8677c-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:35.936145 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.936001    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c6d9f02-368c-4f5e-88fa-789788c8677c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:35.937591 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.937566    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c6d9f02-368c-4f5e-88fa-789788c8677c-proxy-tls" pod "4c6d9f02-368c-4f5e-88fa-789788c8677c" (UID: "4c6d9f02-368c-4f5e-88fa-789788c8677c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:35.937943 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:35.937926    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6d9f02-368c-4f5e-88fa-789788c8677c-kube-api-access-szx48" pod "4c6d9f02-368c-4f5e-88fa-789788c8677c" (UID: "4c6d9f02-368c-4f5e-88fa-789788c8677c"). InnerVolumeSpecName "kube-api-access-szx48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:36.037119 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:36.037043    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c6d9f02-368c-4f5e-88fa-789788c8677c-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:36.037119 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:36.037068    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szx48\" (UniqueName: \"kubernetes.io/projected/4c6d9f02-368c-4f5e-88fa-789788c8677c-kube-api-access-szx48\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:36.476688 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:36.476650    4473 generic.go:356] "Generic (PLEG): container finished" podID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerID="3c540caef259a470c2663d04b4da76420f8853cae69286e0c662fc314cf9509d" exitCode=0
Oct 30 22:49:36.476878 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:36.476708    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"
Oct 30 22:49:36.476878 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:36.476742    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" event={"ID":"4c6d9f02-368c-4f5e-88fa-789788c8677c","Type":"ContainerDied","Data":"3c540caef259a470c2663d04b4da76420f8853cae69286e0c662fc314cf9509d"}
Oct 30 22:49:36.476878 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:36.476803    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9" event={"ID":"4c6d9f02-368c-4f5e-88fa-789788c8677c","Type":"ContainerDied","Data":"954a3a688f15b6b04a7158c353122bfb599dce184685dc1eb74ebacbc5d59f73"}
Oct 30 22:49:36.476878 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:36.476829    4473 scope.go:122] "RemoveContainer" containerID="e86d060b4b096721c2b29eacdddd4b7912e3916c3eb712fcbf35117c5ffef14b"
Oct 30 22:49:36.485606 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:36.485583    4473 scope.go:122] "RemoveContainer" containerID="3c540caef259a470c2663d04b4da76420f8853cae69286e0c662fc314cf9509d"
Oct 30 22:49:36.495597 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:36.495576    4473 scope.go:122] "RemoveContainer" containerID="1cc308c327b074bbfdbb2707d100eeb4e818c3793b5b23a70e6eba255865d6d0"
Oct 30 22:49:36.503280 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:36.503254    4473 scope.go:122] "RemoveContainer" containerID="e86d060b4b096721c2b29eacdddd4b7912e3916c3eb712fcbf35117c5ffef14b"
Oct 30 22:49:36.503595 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:49:36.503575    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86d060b4b096721c2b29eacdddd4b7912e3916c3eb712fcbf35117c5ffef14b\": container with ID starting with e86d060b4b096721c2b29eacdddd4b7912e3916c3eb712fcbf35117c5ffef14b not found: ID does not exist" containerID="e86d060b4b096721c2b29eacdddd4b7912e3916c3eb712fcbf35117c5ffef14b"
Oct 30 22:49:36.503650 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:36.503606    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86d060b4b096721c2b29eacdddd4b7912e3916c3eb712fcbf35117c5ffef14b"} err="failed to get container status \"e86d060b4b096721c2b29eacdddd4b7912e3916c3eb712fcbf35117c5ffef14b\": rpc error: code = NotFound desc = could not find container \"e86d060b4b096721c2b29eacdddd4b7912e3916c3eb712fcbf35117c5ffef14b\": container with ID starting with e86d060b4b096721c2b29eacdddd4b7912e3916c3eb712fcbf35117c5ffef14b not found: ID does not exist"
Oct 30 22:49:36.503650 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:36.503625    4473 scope.go:122] "RemoveContainer" containerID="3c540caef259a470c2663d04b4da76420f8853cae69286e0c662fc314cf9509d"
Oct 30 22:49:36.503864 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:49:36.503848    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c540caef259a470c2663d04b4da76420f8853cae69286e0c662fc314cf9509d\": container with ID starting with 3c540caef259a470c2663d04b4da76420f8853cae69286e0c662fc314cf9509d not found: ID does not exist" containerID="3c540caef259a470c2663d04b4da76420f8853cae69286e0c662fc314cf9509d"
Oct 30 22:49:36.503919 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:36.503869    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c540caef259a470c2663d04b4da76420f8853cae69286e0c662fc314cf9509d"} err="failed to get container status \"3c540caef259a470c2663d04b4da76420f8853cae69286e0c662fc314cf9509d\": rpc error: code = NotFound desc = could not find container \"3c540caef259a470c2663d04b4da76420f8853cae69286e0c662fc314cf9509d\": container with ID starting with 3c540caef259a470c2663d04b4da76420f8853cae69286e0c662fc314cf9509d not found: ID does not exist"
Oct 30 22:49:36.503919 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:36.503884    4473 scope.go:122] "RemoveContainer" containerID="1cc308c327b074bbfdbb2707d100eeb4e818c3793b5b23a70e6eba255865d6d0"
Oct 30 22:49:36.504468 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:49:36.504454    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cc308c327b074bbfdbb2707d100eeb4e818c3793b5b23a70e6eba255865d6d0\": container with ID starting with 1cc308c327b074bbfdbb2707d100eeb4e818c3793b5b23a70e6eba255865d6d0 not found: ID does not exist" containerID="1cc308c327b074bbfdbb2707d100eeb4e818c3793b5b23a70e6eba255865d6d0"
Oct 30 22:49:36.504511 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:36.504473    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cc308c327b074bbfdbb2707d100eeb4e818c3793b5b23a70e6eba255865d6d0"} err="failed to get container status \"1cc308c327b074bbfdbb2707d100eeb4e818c3793b5b23a70e6eba255865d6d0\": rpc error: code = NotFound desc = could not find container \"1cc308c327b074bbfdbb2707d100eeb4e818c3793b5b23a70e6eba255865d6d0\": container with ID starting with 1cc308c327b074bbfdbb2707d100eeb4e818c3793b5b23a70e6eba255865d6d0 not found: ID does not exist"
Oct 30 22:49:36.644301 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:36.644268    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"]
Oct 30 22:49:36.694757 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:36.694714    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7476d02544/isvc-sklearn-s3-tls-serving-pass-predictor-68db6d69bf-5rcz9"]
Oct 30 22:49:37.890809 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:37.887240    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" path="/var/lib/kubelet/pods/4c6d9f02-368c-4f5e-88fa-789788c8677c/volumes"
Oct 30 22:49:38.001315 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:38.001273    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.70:8643/healthz\": dial tcp 10.134.0.70:8643: connect: connection refused"
Oct 30 22:49:40.250874 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:40.250838    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:40.495851 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:40.495818    4473 generic.go:356] "Generic (PLEG): container finished" podID="6a61e24d-5452-468d-b771-26595252b597" containerID="ee2ba74b1c8549ba8e9e59d2c5fbee179f0835d15c41e77200cc75d38e703bc9" exitCode=0
Oct 30 22:49:40.496028 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:40.495889    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" event={"ID":"6a61e24d-5452-468d-b771-26595252b597","Type":"ContainerDied","Data":"ee2ba74b1c8549ba8e9e59d2c5fbee179f0835d15c41e77200cc75d38e703bc9"}
Oct 30 22:49:40.852600 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:40.852576    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:49:40.982402 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:40.982368    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a61e24d-5452-468d-b771-26595252b597-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a61e24d-5452-468d-b771-26595252b597-kserve-provision-location\") pod \"6a61e24d-5452-468d-b771-26595252b597\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") "
Oct 30 22:49:40.982579 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:40.982418    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a61e24d-5452-468d-b771-26595252b597-proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a61e24d-5452-468d-b771-26595252b597-proxy-tls\") pod \"6a61e24d-5452-468d-b771-26595252b597\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") "
Oct 30 22:49:40.982579 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:40.982447    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a61e24d-5452-468d-b771-26595252b597-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a61e24d-5452-468d-b771-26595252b597-kube-rbac-proxy-sar-config\") pod \"6a61e24d-5452-468d-b771-26595252b597\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") "
Oct 30 22:49:40.982579 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:40.982497    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a61e24d-5452-468d-b771-26595252b597-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a61e24d-5452-468d-b771-26595252b597-cabundle-cert\") pod \"6a61e24d-5452-468d-b771-26595252b597\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") "
Oct 30 22:49:40.982579 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:40.982542    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a61e24d-5452-468d-b771-26595252b597-kube-api-access-zq8hx\" (UniqueName: \"kubernetes.io/projected/6a61e24d-5452-468d-b771-26595252b597-kube-api-access-zq8hx\") pod \"6a61e24d-5452-468d-b771-26595252b597\" (UID: \"6a61e24d-5452-468d-b771-26595252b597\") "
Oct 30 22:49:40.982796 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:40.982742    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a61e24d-5452-468d-b771-26595252b597-kserve-provision-location" pod "6a61e24d-5452-468d-b771-26595252b597" (UID: "6a61e24d-5452-468d-b771-26595252b597"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:40.982894 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:40.982871    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a61e24d-5452-468d-b771-26595252b597-kube-rbac-proxy-sar-config" pod "6a61e24d-5452-468d-b771-26595252b597" (UID: "6a61e24d-5452-468d-b771-26595252b597"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:40.982894 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:40.982883    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a61e24d-5452-468d-b771-26595252b597-cabundle-cert" pod "6a61e24d-5452-468d-b771-26595252b597" (UID: "6a61e24d-5452-468d-b771-26595252b597"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:40.984832 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:40.984803    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a61e24d-5452-468d-b771-26595252b597-proxy-tls" pod "6a61e24d-5452-468d-b771-26595252b597" (UID: "6a61e24d-5452-468d-b771-26595252b597"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:40.984917 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:40.984865    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a61e24d-5452-468d-b771-26595252b597-kube-api-access-zq8hx" pod "6a61e24d-5452-468d-b771-26595252b597" (UID: "6a61e24d-5452-468d-b771-26595252b597"). InnerVolumeSpecName "kube-api-access-zq8hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:41.083707 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:41.083623    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a61e24d-5452-468d-b771-26595252b597-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:41.083707 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:41.083656    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a61e24d-5452-468d-b771-26595252b597-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:41.083707 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:41.083667    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a61e24d-5452-468d-b771-26595252b597-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:41.083707 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:41.083676    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a61e24d-5452-468d-b771-26595252b597-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:41.083707 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:41.083685    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq8hx\" (UniqueName: \"kubernetes.io/projected/6a61e24d-5452-468d-b771-26595252b597-kube-api-access-zq8hx\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:41.303043 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:49:41.303008    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:49:41.501882 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:41.501840    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz" event={"ID":"6a61e24d-5452-468d-b771-26595252b597","Type":"ContainerDied","Data":"6cd3dc76e3be11b53a6d3f7821987d95792d1212e673707e3539d4ef87363aff"}
Oct 30 22:49:41.501882 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:41.501871    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"
Oct 30 22:49:41.502165 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:41.501905    4473 scope.go:122] "RemoveContainer" containerID="062ab780807392e803119be77e1bf481cbf32fb89436e3aa67cb35d667681162"
Oct 30 22:49:41.510676 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:41.510657    4473 scope.go:122] "RemoveContainer" containerID="ee2ba74b1c8549ba8e9e59d2c5fbee179f0835d15c41e77200cc75d38e703bc9"
Oct 30 22:49:41.518152 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:41.518134    4473 scope.go:122] "RemoveContainer" containerID="4e58fa18e44be5ffb6f1ceb1f453a37112d17b0f9fc48e0a4f3f1e88d9aa6b5b"
Oct 30 22:49:41.671070 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:41.671032    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"]
Oct 30 22:49:41.719392 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:41.719355    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-lightgbm-predictor-856cd7659f-grvnz"]
Oct 30 22:49:41.886644 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:41.886571    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a61e24d-5452-468d-b771-26595252b597" path="/var/lib/kubelet/pods/6a61e24d-5452-468d-b771-26595252b597/volumes"
Oct 30 22:49:44.782657 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:44.782626    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:45.143255 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:45.143100    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:46.064904 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:46.064877    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:46.251061 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:46.251032    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:51.260066 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:49:51.260021    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:10.556020 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:50:10.555943    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:11.302927 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:50:11.302889    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:50:41.303746 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:50:41.303706    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:51:06.976530 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:06.976502    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:10.887433 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:10.887365    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:11.303740 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:51:11.303702    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:51:13.492794 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:13.492762    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:13.782248 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:13.782160    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:41.303620 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:51:41.303579    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:51:48.739851 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:48.739817    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:57.204350 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.204310    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:57.478235 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.478154    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:57.697481 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.697451    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"]
Oct 30 22:51:57.697760 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.697748    4473 cpu_manager.go:414] "containerMap: removing container" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="storage-initializer"
Oct 30 22:51:57.697807 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.697762    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="storage-initializer"
Oct 30 22:51:57.697807 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.697783    4473 cpu_manager.go:414] "containerMap: removing container" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kube-rbac-proxy"
Oct 30 22:51:57.697807 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.697788    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kube-rbac-proxy"
Oct 30 22:51:57.697807 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.697797    4473 cpu_manager.go:414] "containerMap: removing container" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kube-rbac-proxy"
Oct 30 22:51:57.697807 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.697803    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kube-rbac-proxy"
Oct 30 22:51:57.697980 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.697813    4473 cpu_manager.go:414] "containerMap: removing container" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kserve-container"
Oct 30 22:51:57.697980 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.697818    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kserve-container"
Oct 30 22:51:57.697980 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.697831    4473 cpu_manager.go:414] "containerMap: removing container" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="storage-initializer"
Oct 30 22:51:57.697980 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.697835    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="storage-initializer"
Oct 30 22:51:57.697980 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.697842    4473 cpu_manager.go:414] "containerMap: removing container" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kserve-container"
Oct 30 22:51:57.697980 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.697846    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kserve-container"
Oct 30 22:51:57.697980 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.697894    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kserve-container"
Oct 30 22:51:57.697980 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.697905    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a61e24d-5452-468d-b771-26595252b597" containerName="kube-rbac-proxy"
Oct 30 22:51:57.697980 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.697911    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kube-rbac-proxy"
Oct 30 22:51:57.697980 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.697918    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c6d9f02-368c-4f5e-88fa-789788c8677c" containerName="kserve-container"
Oct 30 22:51:57.701132 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.701112    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:51:57.752513 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.752440    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:51:57.752513 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.752483    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:51:57.752513 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.752505    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:51:57.752730 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.752639    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:51:57.752730 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.752677    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lflgv\" (UniqueName: \"kubernetes.io/projected/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kube-api-access-lflgv\") pod \"isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:51:57.754182 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.754153    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"kube-root-ca.crt\""
Oct 30 22:51:57.754299 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.754193    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 22:51:57.754299 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.754238    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"openshift-service-ca.crt\""
Oct 30 22:51:57.804770 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.804739    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:51:57.804770 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.804757    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"seaweedfs-s3-creds\""
Oct 30 22:51:57.805023 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.804766    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"default-dockercfg-4v4s8\""
Oct 30 22:51:57.805023 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.804794    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Oct 30 22:51:57.806459 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.806438    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"]
Oct 30 22:51:57.853560 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.853534    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:51:57.853696 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.853570    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:51:57.853696 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.853611    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:51:57.853696 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.853635    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lflgv\" (UniqueName: \"kubernetes.io/projected/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kube-api-access-lflgv\") pod \"isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:51:57.853696 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.853680    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:51:57.854155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.854139    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:51:57.854502 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.854479    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:51:57.854546 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.854492    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:51:57.856273 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.856258    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:51:57.912467 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:57.912437    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lflgv\" (UniqueName: \"kubernetes.io/projected/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kube-api-access-lflgv\") pod \"isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") " pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:51:58.017360 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:58.017281    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:51:58.441565 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:58.441540    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"]
Oct 30 22:51:58.443581 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:51:58.443554    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e10f53_078d_4f89_98ab_a8bef5b30c2d.slice/crio-6f1858b184c100bd30aa529ad1df2b4c29101087ce55a3a18800ec5150874953 WatchSource:0}: Error finding container 6f1858b184c100bd30aa529ad1df2b4c29101087ce55a3a18800ec5150874953: Status 404 returned error can't find the container with id 6f1858b184c100bd30aa529ad1df2b4c29101087ce55a3a18800ec5150874953
Oct 30 22:51:59.062131 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:59.062094    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs" event={"ID":"c6e10f53-078d-4f89-98ab-a8bef5b30c2d","Type":"ContainerStarted","Data":"de158fd90afff67461c4eebb1599b668974330a122064ae280cd2703f4d4c7f7"}
Oct 30 22:51:59.062308 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:51:59.062153    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs" event={"ID":"c6e10f53-078d-4f89-98ab-a8bef5b30c2d","Type":"ContainerStarted","Data":"6f1858b184c100bd30aa529ad1df2b4c29101087ce55a3a18800ec5150874953"}
Oct 30 22:52:03.079504 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:03.079463    4473 generic.go:356] "Generic (PLEG): container finished" podID="c6e10f53-078d-4f89-98ab-a8bef5b30c2d" containerID="de158fd90afff67461c4eebb1599b668974330a122064ae280cd2703f4d4c7f7" exitCode=0
Oct 30 22:52:03.079867 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:03.079534    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs" event={"ID":"c6e10f53-078d-4f89-98ab-a8bef5b30c2d","Type":"ContainerDied","Data":"de158fd90afff67461c4eebb1599b668974330a122064ae280cd2703f4d4c7f7"}
Oct 30 22:52:04.086444 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:04.086410    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs" event={"ID":"c6e10f53-078d-4f89-98ab-a8bef5b30c2d","Type":"ContainerStarted","Data":"f15b73e94a252e593d2dd3c5c37d1d4b0e8b98d005bf71a65f2c4fd337bbbee0"}
Oct 30 22:52:04.086885 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:04.086471    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs" event={"ID":"c6e10f53-078d-4f89-98ab-a8bef5b30c2d","Type":"ContainerStarted","Data":"f5c5663e8f863a196b110d1a325681e7a4ea875a8c49375dafc103a371492e5d"}
Oct 30 22:52:04.086885 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:04.086697    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:52:04.086885 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:04.086876    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:52:04.256922 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:04.256794    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs" podStartSLOduration=3.408052645 podStartE2EDuration="7.256776729s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.848724084s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:04.256385843 +0000 UTC m=+3542.492595189" watchObservedRunningTime="2026-10-30 22:52:04.256776729 +0000 UTC m=+3542.492986072"
Oct 30 22:52:10.096802 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:10.096763    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:52:11.263969 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:11.263927    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:11.302566 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:52:11.302533    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:52:15.676249 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:15.676182    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:22.203103 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:22.203066    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:22.412139 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:22.412109    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:40.101327 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:40.101291    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:52:41.303348 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:52:41.303312    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:52:47.842123 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:47.842087    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"]
Oct 30 22:52:47.842581 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:47.842312    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs" podUID="c6e10f53-078d-4f89-98ab-a8bef5b30c2d" containerName="kserve-container" containerID="cri-o://f5c5663e8f863a196b110d1a325681e7a4ea875a8c49375dafc103a371492e5d" gracePeriod=30
Oct 30 22:52:47.842581 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:47.842363    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs" podUID="c6e10f53-078d-4f89-98ab-a8bef5b30c2d" containerName="kube-rbac-proxy" containerID="cri-o://f15b73e94a252e593d2dd3c5c37d1d4b0e8b98d005bf71a65f2c4fd337bbbee0" gracePeriod=30
Oct 30 22:52:47.861203 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:47.861172    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:48.272489 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:48.272455    4473 generic.go:356] "Generic (PLEG): container finished" podID="c6e10f53-078d-4f89-98ab-a8bef5b30c2d" containerID="f15b73e94a252e593d2dd3c5c37d1d4b0e8b98d005bf71a65f2c4fd337bbbee0" exitCode=0
Oct 30 22:52:48.272668 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:48.272530    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs" event={"ID":"c6e10f53-078d-4f89-98ab-a8bef5b30c2d","Type":"ContainerDied","Data":"f15b73e94a252e593d2dd3c5c37d1d4b0e8b98d005bf71a65f2c4fd337bbbee0"}
Oct 30 22:52:48.882228 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:48.882204    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:52:48.986743 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:48.986650    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-cabundle-cert\") pod \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") "
Oct 30 22:52:48.986743 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:48.986687    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kserve-provision-location\") pod \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") "
Oct 30 22:52:48.987011 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:48.986752    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kube-api-access-lflgv\" (UniqueName: \"kubernetes.io/projected/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kube-api-access-lflgv\") pod \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") "
Oct 30 22:52:48.987011 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:48.986772    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kube-rbac-proxy-sar-config\") pod \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") "
Oct 30 22:52:48.987011 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:48.986807    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-proxy-tls\") pod \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\" (UID: \"c6e10f53-078d-4f89-98ab-a8bef5b30c2d\") "
Oct 30 22:52:48.987187 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:48.987112    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-cabundle-cert" pod "c6e10f53-078d-4f89-98ab-a8bef5b30c2d" (UID: "c6e10f53-078d-4f89-98ab-a8bef5b30c2d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:48.987187 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:48.987142    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kserve-provision-location" pod "c6e10f53-078d-4f89-98ab-a8bef5b30c2d" (UID: "c6e10f53-078d-4f89-98ab-a8bef5b30c2d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:48.987187 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:48.987176    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kube-rbac-proxy-sar-config" pod "c6e10f53-078d-4f89-98ab-a8bef5b30c2d" (UID: "c6e10f53-078d-4f89-98ab-a8bef5b30c2d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:48.989103 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:48.989079    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-proxy-tls" pod "c6e10f53-078d-4f89-98ab-a8bef5b30c2d" (UID: "c6e10f53-078d-4f89-98ab-a8bef5b30c2d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:48.989164 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:48.989141    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kube-api-access-lflgv" pod "c6e10f53-078d-4f89-98ab-a8bef5b30c2d" (UID: "c6e10f53-078d-4f89-98ab-a8bef5b30c2d"). InnerVolumeSpecName "kube-api-access-lflgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:49.087754 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.087724    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:49.087754 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.087748    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:49.087754 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.087759    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:49.088001 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.087771    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lflgv\" (UniqueName: \"kubernetes.io/projected/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kube-api-access-lflgv\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:49.088001 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.087781    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6e10f53-078d-4f89-98ab-a8bef5b30c2d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:49.278512 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.278420    4473 generic.go:356] "Generic (PLEG): container finished" podID="c6e10f53-078d-4f89-98ab-a8bef5b30c2d" containerID="f5c5663e8f863a196b110d1a325681e7a4ea875a8c49375dafc103a371492e5d" exitCode=0
Oct 30 22:52:49.278512 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.278487    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"
Oct 30 22:52:49.278512 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.278488    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs" event={"ID":"c6e10f53-078d-4f89-98ab-a8bef5b30c2d","Type":"ContainerDied","Data":"f5c5663e8f863a196b110d1a325681e7a4ea875a8c49375dafc103a371492e5d"}
Oct 30 22:52:49.278740 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.278550    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs" event={"ID":"c6e10f53-078d-4f89-98ab-a8bef5b30c2d","Type":"ContainerDied","Data":"6f1858b184c100bd30aa529ad1df2b4c29101087ce55a3a18800ec5150874953"}
Oct 30 22:52:49.278740 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.278577    4473 scope.go:122] "RemoveContainer" containerID="f15b73e94a252e593d2dd3c5c37d1d4b0e8b98d005bf71a65f2c4fd337bbbee0"
Oct 30 22:52:49.287473 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.287456    4473 scope.go:122] "RemoveContainer" containerID="f5c5663e8f863a196b110d1a325681e7a4ea875a8c49375dafc103a371492e5d"
Oct 30 22:52:49.294761 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.294736    4473 scope.go:122] "RemoveContainer" containerID="de158fd90afff67461c4eebb1599b668974330a122064ae280cd2703f4d4c7f7"
Oct 30 22:52:49.301740 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.301724    4473 scope.go:122] "RemoveContainer" containerID="f15b73e94a252e593d2dd3c5c37d1d4b0e8b98d005bf71a65f2c4fd337bbbee0"
Oct 30 22:52:49.302021 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:52:49.302001    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f15b73e94a252e593d2dd3c5c37d1d4b0e8b98d005bf71a65f2c4fd337bbbee0\": container with ID starting with f15b73e94a252e593d2dd3c5c37d1d4b0e8b98d005bf71a65f2c4fd337bbbee0 not found: ID does not exist" containerID="f15b73e94a252e593d2dd3c5c37d1d4b0e8b98d005bf71a65f2c4fd337bbbee0"
Oct 30 22:52:49.302081 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.302031    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f15b73e94a252e593d2dd3c5c37d1d4b0e8b98d005bf71a65f2c4fd337bbbee0"} err="failed to get container status \"f15b73e94a252e593d2dd3c5c37d1d4b0e8b98d005bf71a65f2c4fd337bbbee0\": rpc error: code = NotFound desc = could not find container \"f15b73e94a252e593d2dd3c5c37d1d4b0e8b98d005bf71a65f2c4fd337bbbee0\": container with ID starting with f15b73e94a252e593d2dd3c5c37d1d4b0e8b98d005bf71a65f2c4fd337bbbee0 not found: ID does not exist"
Oct 30 22:52:49.302081 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.302050    4473 scope.go:122] "RemoveContainer" containerID="f5c5663e8f863a196b110d1a325681e7a4ea875a8c49375dafc103a371492e5d"
Oct 30 22:52:49.302295 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:52:49.302279    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c5663e8f863a196b110d1a325681e7a4ea875a8c49375dafc103a371492e5d\": container with ID starting with f5c5663e8f863a196b110d1a325681e7a4ea875a8c49375dafc103a371492e5d not found: ID does not exist" containerID="f5c5663e8f863a196b110d1a325681e7a4ea875a8c49375dafc103a371492e5d"
Oct 30 22:52:49.302329 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.302303    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c5663e8f863a196b110d1a325681e7a4ea875a8c49375dafc103a371492e5d"} err="failed to get container status \"f5c5663e8f863a196b110d1a325681e7a4ea875a8c49375dafc103a371492e5d\": rpc error: code = NotFound desc = could not find container \"f5c5663e8f863a196b110d1a325681e7a4ea875a8c49375dafc103a371492e5d\": container with ID starting with f5c5663e8f863a196b110d1a325681e7a4ea875a8c49375dafc103a371492e5d not found: ID does not exist"
Oct 30 22:52:49.302329 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.302317    4473 scope.go:122] "RemoveContainer" containerID="de158fd90afff67461c4eebb1599b668974330a122064ae280cd2703f4d4c7f7"
Oct 30 22:52:49.302571 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:52:49.302557    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de158fd90afff67461c4eebb1599b668974330a122064ae280cd2703f4d4c7f7\": container with ID starting with de158fd90afff67461c4eebb1599b668974330a122064ae280cd2703f4d4c7f7 not found: ID does not exist" containerID="de158fd90afff67461c4eebb1599b668974330a122064ae280cd2703f4d4c7f7"
Oct 30 22:52:49.302606 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.302578    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de158fd90afff67461c4eebb1599b668974330a122064ae280cd2703f4d4c7f7"} err="failed to get container status \"de158fd90afff67461c4eebb1599b668974330a122064ae280cd2703f4d4c7f7\": rpc error: code = NotFound desc = could not find container \"de158fd90afff67461c4eebb1599b668974330a122064ae280cd2703f4d4c7f7\": container with ID starting with de158fd90afff67461c4eebb1599b668974330a122064ae280cd2703f4d4c7f7 not found: ID does not exist"
Oct 30 22:52:49.445659 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.445625    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"]
Oct 30 22:52:49.447419 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.447398    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-lightgbm-v2-runtime-predictor-7fd7559bbc-pfsjs"]
Oct 30 22:52:49.886347 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:49.886313    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6e10f53-078d-4f89-98ab-a8bef5b30c2d" path="/var/lib/kubelet/pods/c6e10f53-078d-4f89-98ab-a8bef5b30c2d/volumes"
Oct 30 22:52:52.363598 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:52.363560    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:52.606195 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:52:52.606160    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:02.216924 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:02.216814    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:53:02.224062 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:02.224042    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:53:11.303571 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:53:11.303535    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:53:11.570222 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:11.570144    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:23.015908 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:23.015847    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:24.559934 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:24.559898    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:27.471468 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:27.471431    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:27.810654 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:27.810572    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:27.966470 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:27.966427    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"]
Oct 30 22:53:27.966757 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:27.966744    4473 cpu_manager.go:414] "containerMap: removing container" podUID="c6e10f53-078d-4f89-98ab-a8bef5b30c2d" containerName="storage-initializer"
Oct 30 22:53:27.966804 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:27.966758    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6e10f53-078d-4f89-98ab-a8bef5b30c2d" containerName="storage-initializer"
Oct 30 22:53:27.966804 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:27.966779    4473 cpu_manager.go:414] "containerMap: removing container" podUID="c6e10f53-078d-4f89-98ab-a8bef5b30c2d" containerName="kube-rbac-proxy"
Oct 30 22:53:27.966804 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:27.966784    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6e10f53-078d-4f89-98ab-a8bef5b30c2d" containerName="kube-rbac-proxy"
Oct 30 22:53:27.966804 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:27.966799    4473 cpu_manager.go:414] "containerMap: removing container" podUID="c6e10f53-078d-4f89-98ab-a8bef5b30c2d" containerName="kserve-container"
Oct 30 22:53:27.966804 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:27.966804    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6e10f53-078d-4f89-98ab-a8bef5b30c2d" containerName="kserve-container"
Oct 30 22:53:27.966987 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:27.966857    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6e10f53-078d-4f89-98ab-a8bef5b30c2d" containerName="kube-rbac-proxy"
Oct 30 22:53:27.966987 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:27.966867    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6e10f53-078d-4f89-98ab-a8bef5b30c2d" containerName="kserve-container"
Oct 30 22:53:27.969962 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:27.969936    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:28.014673 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.014630    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d012bd9-6653-45a6-b7d6-e54f65f929a3-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") " pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:28.014834 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.014682    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") " pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:28.014834 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.014768    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") " pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:28.014834 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.014790    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d012bd9-6653-45a6-b7d6-e54f65f929a3-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") " pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:28.014834 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.014810    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9wqq\" (UniqueName: \"kubernetes.io/projected/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kube-api-access-w9wqq\") pod \"isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") " pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:28.022963 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.022933    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"seaweedfs-s3-creds\""
Oct 30 22:53:28.023155 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.023138    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"default-dockercfg-4v4s8\""
Oct 30 22:53:28.023426 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.023404    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:53:28.073610 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.073533    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Oct 30 22:53:28.073610 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.073544    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"kube-root-ca.crt\""
Oct 30 22:53:28.073797 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.073753    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"openshift-service-ca.crt\""
Oct 30 22:53:28.073797 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.073772    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-37d062d1dd\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 22:53:28.075357 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.075340    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"]
Oct 30 22:53:28.115189 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.115155    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") " pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:28.115351 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.115193    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d012bd9-6653-45a6-b7d6-e54f65f929a3-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") " pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:28.115351 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.115216    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9wqq\" (UniqueName: \"kubernetes.io/projected/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kube-api-access-w9wqq\") pod \"isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") " pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:28.115351 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.115265    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d012bd9-6653-45a6-b7d6-e54f65f929a3-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") " pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:28.115351 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.115287    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") " pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:28.115750 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.115714    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") " pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:28.115905 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.115888    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d012bd9-6653-45a6-b7d6-e54f65f929a3-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") " pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:28.115945 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.115898    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") " pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:28.117759 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.117741    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d012bd9-6653-45a6-b7d6-e54f65f929a3-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") " pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:28.173909 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.173881    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9wqq\" (UniqueName: \"kubernetes.io/projected/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kube-api-access-w9wqq\") pod \"isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") " pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:28.288924 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.288894    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:28.718221 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:28.718180    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"]
Oct 30 22:53:28.757781 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:53:28.757753    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d012bd9_6653_45a6_b7d6_e54f65f929a3.slice/crio-53c671db0f6f4012ababd426b74ca966e01e697ebfe89036c7f599614dd4d102 WatchSource:0}: Error finding container 53c671db0f6f4012ababd426b74ca966e01e697ebfe89036c7f599614dd4d102: Status 404 returned error can't find the container with id 53c671db0f6f4012ababd426b74ca966e01e697ebfe89036c7f599614dd4d102
Oct 30 22:53:29.444449 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:29.444409    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2" event={"ID":"8d012bd9-6653-45a6-b7d6-e54f65f929a3","Type":"ContainerStarted","Data":"5552c4d21bf49d2bf5ed5f7c7f19e7114442c8139dc6175c0f23bbc90ff9ed91"}
Oct 30 22:53:29.444627 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:29.444485    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2" event={"ID":"8d012bd9-6653-45a6-b7d6-e54f65f929a3","Type":"ContainerStarted","Data":"53c671db0f6f4012ababd426b74ca966e01e697ebfe89036c7f599614dd4d102"}
Oct 30 22:53:32.300854 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:32.300822    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:32.483097 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:32.483060    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:33.462033 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:33.462000    4473 generic.go:356] "Generic (PLEG): container finished" podID="8d012bd9-6653-45a6-b7d6-e54f65f929a3" containerID="5552c4d21bf49d2bf5ed5f7c7f19e7114442c8139dc6175c0f23bbc90ff9ed91" exitCode=0
Oct 30 22:53:33.462510 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:33.462071    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2" event={"ID":"8d012bd9-6653-45a6-b7d6-e54f65f929a3","Type":"ContainerDied","Data":"5552c4d21bf49d2bf5ed5f7c7f19e7114442c8139dc6175c0f23bbc90ff9ed91"}
Oct 30 22:53:34.469449 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:34.469417    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2" event={"ID":"8d012bd9-6653-45a6-b7d6-e54f65f929a3","Type":"ContainerStarted","Data":"e2b41634ec44c5c2b72871ad47f2e4a37f20dcde154170ab7abe09e0ba94cca3"}
Oct 30 22:53:34.469980 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:34.469483    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2" event={"ID":"8d012bd9-6653-45a6-b7d6-e54f65f929a3","Type":"ContainerStarted","Data":"e1f00af628b675cb6bcf5670005d8bed70a95b4f0fbfef01a1f3d202c72224a9"}
Oct 30 22:53:34.469980 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:34.469805    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:34.469980 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:34.469861    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:34.642002 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:34.641911    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2" podStartSLOduration=3.722479326 podStartE2EDuration="7.641890078s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.919410752s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:34.590522973 +0000 UTC m=+3632.826732318" watchObservedRunningTime="2026-10-30 22:53:34.641890078 +0000 UTC m=+3632.878099428"
Oct 30 22:53:40.480267 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:53:40.480237    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:53:41.303406 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:53:41.303369    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:54:10.484276 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:10.484243    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:54:11.304048 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:54:11.304014    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:54:11.993234 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:11.993197    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:18.121470 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:18.121437    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"]
Oct 30 22:54:18.121971 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:18.121663    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2" podUID="8d012bd9-6653-45a6-b7d6-e54f65f929a3" containerName="kserve-container" containerID="cri-o://e1f00af628b675cb6bcf5670005d8bed70a95b4f0fbfef01a1f3d202c72224a9" gracePeriod=30
Oct 30 22:54:18.121971 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:18.121744    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2" podUID="8d012bd9-6653-45a6-b7d6-e54f65f929a3" containerName="kube-rbac-proxy" containerID="cri-o://e2b41634ec44c5c2b72871ad47f2e4a37f20dcde154170ab7abe09e0ba94cca3" gracePeriod=30
Oct 30 22:54:18.130756 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:18.130722    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:18.654293 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:18.654262    4473 generic.go:356] "Generic (PLEG): container finished" podID="8d012bd9-6653-45a6-b7d6-e54f65f929a3" containerID="e2b41634ec44c5c2b72871ad47f2e4a37f20dcde154170ab7abe09e0ba94cca3" exitCode=0
Oct 30 22:54:18.654520 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:18.654345    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2" event={"ID":"8d012bd9-6653-45a6-b7d6-e54f65f929a3","Type":"ContainerDied","Data":"e2b41634ec44c5c2b72871ad47f2e4a37f20dcde154170ab7abe09e0ba94cca3"}
Oct 30 22:54:19.370234 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.370210    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:54:19.538929 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.538847    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kube-rbac-proxy-sar-config\") pod \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") "
Oct 30 22:54:19.538929 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.538908    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kserve-provision-location\") pod \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") "
Oct 30 22:54:19.539196 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.538940    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d012bd9-6653-45a6-b7d6-e54f65f929a3-proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d012bd9-6653-45a6-b7d6-e54f65f929a3-proxy-tls\") pod \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") "
Oct 30 22:54:19.539196 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.538997    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d012bd9-6653-45a6-b7d6-e54f65f929a3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d012bd9-6653-45a6-b7d6-e54f65f929a3-cabundle-cert\") pod \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") "
Oct 30 22:54:19.539196 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.539027    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kube-api-access-w9wqq\" (UniqueName: \"kubernetes.io/projected/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kube-api-access-w9wqq\") pod \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\" (UID: \"8d012bd9-6653-45a6-b7d6-e54f65f929a3\") "
Oct 30 22:54:19.539354 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.539282    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kube-rbac-proxy-sar-config" pod "8d012bd9-6653-45a6-b7d6-e54f65f929a3" (UID: "8d012bd9-6653-45a6-b7d6-e54f65f929a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:19.539354 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.539289    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kserve-provision-location" pod "8d012bd9-6653-45a6-b7d6-e54f65f929a3" (UID: "8d012bd9-6653-45a6-b7d6-e54f65f929a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:19.539354 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.539334    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d012bd9-6653-45a6-b7d6-e54f65f929a3-cabundle-cert" pod "8d012bd9-6653-45a6-b7d6-e54f65f929a3" (UID: "8d012bd9-6653-45a6-b7d6-e54f65f929a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:19.541262 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.541236    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d012bd9-6653-45a6-b7d6-e54f65f929a3-proxy-tls" pod "8d012bd9-6653-45a6-b7d6-e54f65f929a3" (UID: "8d012bd9-6653-45a6-b7d6-e54f65f929a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:19.541373 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.541263    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kube-api-access-w9wqq" pod "8d012bd9-6653-45a6-b7d6-e54f65f929a3" (UID: "8d012bd9-6653-45a6-b7d6-e54f65f929a3"). InnerVolumeSpecName "kube-api-access-w9wqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:19.640360 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.640324    4473 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:19.640360 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.640354    4473 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kserve-provision-location\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:19.640360 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.640366    4473 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d012bd9-6653-45a6-b7d6-e54f65f929a3-proxy-tls\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:19.640603 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.640376    4473 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d012bd9-6653-45a6-b7d6-e54f65f929a3-cabundle-cert\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:19.640603 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.640384    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9wqq\" (UniqueName: \"kubernetes.io/projected/8d012bd9-6653-45a6-b7d6-e54f65f929a3-kube-api-access-w9wqq\") on node \"ip-10-0-130-162.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:19.660601 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.660568    4473 generic.go:356] "Generic (PLEG): container finished" podID="8d012bd9-6653-45a6-b7d6-e54f65f929a3" containerID="e1f00af628b675cb6bcf5670005d8bed70a95b4f0fbfef01a1f3d202c72224a9" exitCode=0
Oct 30 22:54:19.660757 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.660636    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"
Oct 30 22:54:19.660757 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.660657    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2" event={"ID":"8d012bd9-6653-45a6-b7d6-e54f65f929a3","Type":"ContainerDied","Data":"e1f00af628b675cb6bcf5670005d8bed70a95b4f0fbfef01a1f3d202c72224a9"}
Oct 30 22:54:19.660757 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.660724    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2" event={"ID":"8d012bd9-6653-45a6-b7d6-e54f65f929a3","Type":"ContainerDied","Data":"53c671db0f6f4012ababd426b74ca966e01e697ebfe89036c7f599614dd4d102"}
Oct 30 22:54:19.660873 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.660763    4473 scope.go:122] "RemoveContainer" containerID="e2b41634ec44c5c2b72871ad47f2e4a37f20dcde154170ab7abe09e0ba94cca3"
Oct 30 22:54:19.669630 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.669610    4473 scope.go:122] "RemoveContainer" containerID="e1f00af628b675cb6bcf5670005d8bed70a95b4f0fbfef01a1f3d202c72224a9"
Oct 30 22:54:19.677426 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.677406    4473 scope.go:122] "RemoveContainer" containerID="5552c4d21bf49d2bf5ed5f7c7f19e7114442c8139dc6175c0f23bbc90ff9ed91"
Oct 30 22:54:19.684687 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.684668    4473 scope.go:122] "RemoveContainer" containerID="e2b41634ec44c5c2b72871ad47f2e4a37f20dcde154170ab7abe09e0ba94cca3"
Oct 30 22:54:19.684926 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:54:19.684906    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b41634ec44c5c2b72871ad47f2e4a37f20dcde154170ab7abe09e0ba94cca3\": container with ID starting with e2b41634ec44c5c2b72871ad47f2e4a37f20dcde154170ab7abe09e0ba94cca3 not found: ID does not exist" containerID="e2b41634ec44c5c2b72871ad47f2e4a37f20dcde154170ab7abe09e0ba94cca3"
Oct 30 22:54:19.685024 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.684940    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b41634ec44c5c2b72871ad47f2e4a37f20dcde154170ab7abe09e0ba94cca3"} err="failed to get container status \"e2b41634ec44c5c2b72871ad47f2e4a37f20dcde154170ab7abe09e0ba94cca3\": rpc error: code = NotFound desc = could not find container \"e2b41634ec44c5c2b72871ad47f2e4a37f20dcde154170ab7abe09e0ba94cca3\": container with ID starting with e2b41634ec44c5c2b72871ad47f2e4a37f20dcde154170ab7abe09e0ba94cca3 not found: ID does not exist"
Oct 30 22:54:19.685024 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.684991    4473 scope.go:122] "RemoveContainer" containerID="e1f00af628b675cb6bcf5670005d8bed70a95b4f0fbfef01a1f3d202c72224a9"
Oct 30 22:54:19.685263 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:54:19.685245    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f00af628b675cb6bcf5670005d8bed70a95b4f0fbfef01a1f3d202c72224a9\": container with ID starting with e1f00af628b675cb6bcf5670005d8bed70a95b4f0fbfef01a1f3d202c72224a9 not found: ID does not exist" containerID="e1f00af628b675cb6bcf5670005d8bed70a95b4f0fbfef01a1f3d202c72224a9"
Oct 30 22:54:19.685305 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.685272    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f00af628b675cb6bcf5670005d8bed70a95b4f0fbfef01a1f3d202c72224a9"} err="failed to get container status \"e1f00af628b675cb6bcf5670005d8bed70a95b4f0fbfef01a1f3d202c72224a9\": rpc error: code = NotFound desc = could not find container \"e1f00af628b675cb6bcf5670005d8bed70a95b4f0fbfef01a1f3d202c72224a9\": container with ID starting with e1f00af628b675cb6bcf5670005d8bed70a95b4f0fbfef01a1f3d202c72224a9 not found: ID does not exist"
Oct 30 22:54:19.685305 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.685290    4473 scope.go:122] "RemoveContainer" containerID="5552c4d21bf49d2bf5ed5f7c7f19e7114442c8139dc6175c0f23bbc90ff9ed91"
Oct 30 22:54:19.685516 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:54:19.685502    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5552c4d21bf49d2bf5ed5f7c7f19e7114442c8139dc6175c0f23bbc90ff9ed91\": container with ID starting with 5552c4d21bf49d2bf5ed5f7c7f19e7114442c8139dc6175c0f23bbc90ff9ed91 not found: ID does not exist" containerID="5552c4d21bf49d2bf5ed5f7c7f19e7114442c8139dc6175c0f23bbc90ff9ed91"
Oct 30 22:54:19.685559 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.685519    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5552c4d21bf49d2bf5ed5f7c7f19e7114442c8139dc6175c0f23bbc90ff9ed91"} err="failed to get container status \"5552c4d21bf49d2bf5ed5f7c7f19e7114442c8139dc6175c0f23bbc90ff9ed91\": rpc error: code = NotFound desc = could not find container \"5552c4d21bf49d2bf5ed5f7c7f19e7114442c8139dc6175c0f23bbc90ff9ed91\": container with ID starting with 5552c4d21bf49d2bf5ed5f7c7f19e7114442c8139dc6175c0f23bbc90ff9ed91 not found: ID does not exist"
Oct 30 22:54:19.828694 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.828609    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"]
Oct 30 22:54:19.832238 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.832210    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-37d062d1dd/isvc-mlflow-v2-runtime-predictor-68858b965b-6xqb2"]
Oct 30 22:54:19.886062 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:19.886027    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d012bd9-6653-45a6-b7d6-e54f65f929a3" path="/var/lib/kubelet/pods/8d012bd9-6653-45a6-b7d6-e54f65f929a3/volumes"
Oct 30 22:54:22.681571 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:22.681533    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:22.997433 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:54:22.997351    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:41.303611 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:54:41.303573    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:55:11.304058 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:55:11.304017    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:55:12.243128 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:55:12.243048    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:23.835115 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:55:23.835073    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:30.307842 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:55:30.307809    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:30.607038 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:55:30.606931    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:41.304245 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:55:41.304201    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:55:53.121478 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:55:53.121443    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:54.968695 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:55:54.968656    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:11.303615 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:56:11.303571    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:56:12.598393 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:56:12.598359    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:24.668194 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:56:24.668128    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:33.894479 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:56:33.894445    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:34.053281 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:56:34.053247    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:41.303966 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:56:41.303907    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:57:11.304660 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:57:11.304626    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:57:12.823742 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:57:12.823682    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:12.997599 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:57:12.997567    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:17.930579 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:57:17.930545    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:19.696817 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:57:19.696783    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:19.883558 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:57:19.883518    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:25.170304 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:57:25.170269    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:26.321137 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:57:26.321102    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:26.477009 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:57:26.476975    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:41.304588 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:57:41.304547    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:58:02.242221 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:02.242101    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:58:02.249487 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:02.249466    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8xr5_d46f22d3-4503-41af-98d9-c641d343de28/ovn-acl-logging/0.log"
Oct 30 22:58:11.303334 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:58:11.303288    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:58:13.278692 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:13.278616    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:34.912589 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:34.912555    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:35.197647 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.197567    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k2tk6/must-gather-vz7xg"]
Oct 30 22:58:35.197909 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.197896    4473 cpu_manager.go:414] "containerMap: removing container" podUID="8d012bd9-6653-45a6-b7d6-e54f65f929a3" containerName="storage-initializer"
Oct 30 22:58:35.197972 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.197914    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d012bd9-6653-45a6-b7d6-e54f65f929a3" containerName="storage-initializer"
Oct 30 22:58:35.197972 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.197927    4473 cpu_manager.go:414] "containerMap: removing container" podUID="8d012bd9-6653-45a6-b7d6-e54f65f929a3" containerName="kserve-container"
Oct 30 22:58:35.197972 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.197932    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d012bd9-6653-45a6-b7d6-e54f65f929a3" containerName="kserve-container"
Oct 30 22:58:35.198077 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.197973    4473 cpu_manager.go:414] "containerMap: removing container" podUID="8d012bd9-6653-45a6-b7d6-e54f65f929a3" containerName="kube-rbac-proxy"
Oct 30 22:58:35.198077 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.197979    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d012bd9-6653-45a6-b7d6-e54f65f929a3" containerName="kube-rbac-proxy"
Oct 30 22:58:35.198077 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.198037    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d012bd9-6653-45a6-b7d6-e54f65f929a3" containerName="kube-rbac-proxy"
Oct 30 22:58:35.198077 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.198047    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d012bd9-6653-45a6-b7d6-e54f65f929a3" containerName="kserve-container"
Oct 30 22:58:35.201125 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.201110    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k2tk6/must-gather-vz7xg"
Oct 30 22:58:35.204524 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.204492    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:35.231846 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.231817    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxgrv\" (UniqueName: \"kubernetes.io/projected/f596a0b7-077b-435f-aa06-b52156a6995f-kube-api-access-kxgrv\") pod \"must-gather-vz7xg\" (UID: \"f596a0b7-077b-435f-aa06-b52156a6995f\") " pod="openshift-must-gather-k2tk6/must-gather-vz7xg"
Oct 30 22:58:35.231975 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.231854    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f596a0b7-077b-435f-aa06-b52156a6995f-must-gather-output\") pod \"must-gather-vz7xg\" (UID: \"f596a0b7-077b-435f-aa06-b52156a6995f\") " pod="openshift-must-gather-k2tk6/must-gather-vz7xg"
Oct 30 22:58:35.259325 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.259301    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k2tk6/must-gather-vz7xg"]
Oct 30 22:58:35.304689 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.304658    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k2tk6\"/\"default-dockercfg-fqshl\""
Oct 30 22:58:35.304973 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.304931    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k2tk6\"/\"openshift-service-ca.crt\""
Oct 30 22:58:35.305032 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.304987    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k2tk6\"/\"kube-root-ca.crt\""
Oct 30 22:58:35.333218 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.333187    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxgrv\" (UniqueName: \"kubernetes.io/projected/f596a0b7-077b-435f-aa06-b52156a6995f-kube-api-access-kxgrv\") pod \"must-gather-vz7xg\" (UID: \"f596a0b7-077b-435f-aa06-b52156a6995f\") " pod="openshift-must-gather-k2tk6/must-gather-vz7xg"
Oct 30 22:58:35.333367 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.333226    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f596a0b7-077b-435f-aa06-b52156a6995f-must-gather-output\") pod \"must-gather-vz7xg\" (UID: \"f596a0b7-077b-435f-aa06-b52156a6995f\") " pod="openshift-must-gather-k2tk6/must-gather-vz7xg"
Oct 30 22:58:35.333562 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.333543    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f596a0b7-077b-435f-aa06-b52156a6995f-must-gather-output\") pod \"must-gather-vz7xg\" (UID: \"f596a0b7-077b-435f-aa06-b52156a6995f\") " pod="openshift-must-gather-k2tk6/must-gather-vz7xg"
Oct 30 22:58:35.373388 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.373365    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:35.390856 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.390834    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxgrv\" (UniqueName: \"kubernetes.io/projected/f596a0b7-077b-435f-aa06-b52156a6995f-kube-api-access-kxgrv\") pod \"must-gather-vz7xg\" (UID: \"f596a0b7-077b-435f-aa06-b52156a6995f\") " pod="openshift-must-gather-k2tk6/must-gather-vz7xg"
Oct 30 22:58:35.528104 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.528014    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k2tk6/must-gather-vz7xg"
Oct 30 22:58:35.899044 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.899019    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k2tk6/must-gather-vz7xg"]
Oct 30 22:58:35.901539 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:58:35.901515    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf596a0b7_077b_435f_aa06_b52156a6995f.slice/crio-7fb88c28fa01d18086f7506f3bf6b5de81907d9baf0f304633a7c5c3e2dd3743 WatchSource:0}: Error finding container 7fb88c28fa01d18086f7506f3bf6b5de81907d9baf0f304633a7c5c3e2dd3743: Status 404 returned error can't find the container with id 7fb88c28fa01d18086f7506f3bf6b5de81907d9baf0f304633a7c5c3e2dd3743
Oct 30 22:58:35.903335 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:35.903321    4473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:58:36.651964 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:36.651928    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k2tk6/must-gather-vz7xg" event={"ID":"f596a0b7-077b-435f-aa06-b52156a6995f","Type":"ContainerStarted","Data":"7fb88c28fa01d18086f7506f3bf6b5de81907d9baf0f304633a7c5c3e2dd3743"}
Oct 30 22:58:37.657492 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:37.657453    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k2tk6/must-gather-vz7xg" event={"ID":"f596a0b7-077b-435f-aa06-b52156a6995f","Type":"ContainerStarted","Data":"94564a964aa87a2ac0f3434ab3c10c7b5fc071dfc4488cfd5bba5ed43eeb1729"}
Oct 30 22:58:37.658016 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:37.657501    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k2tk6/must-gather-vz7xg" event={"ID":"f596a0b7-077b-435f-aa06-b52156a6995f","Type":"ContainerStarted","Data":"9c7fe6086405b3845b2c9f2eda3d2d0774826a051c3e5210274412d423ae74f9"}
Oct 30 22:58:37.774089 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:37.774041    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k2tk6/must-gather-vz7xg" podStartSLOduration=1.570594993 podStartE2EDuration="2.774026614s" totalImagesPullingTime="1.203431621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:37.772767356 +0000 UTC m=+3936.008976702" watchObservedRunningTime="2026-10-30 22:58:37.774026614 +0000 UTC m=+3936.010236019"
Oct 30 22:58:40.870967 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:40.870901    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:41.049013 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:41.048977    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:41.309241 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:58:41.309204    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:58:44.093312 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:44.093282    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4tlxp_4cced475-7943-49ad-870b-4b7119d62759/global-pull-secret-syncer/0.log"
Oct 30 22:58:47.353012 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:47.352973    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log"
Oct 30 22:58:48.855478 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:48.855436    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-162.us-west-1.compute.internal_a49dc1ff1f4db21ad91c65b8f36133e5/haproxy/0.log"
Oct 30 22:58:57.389550 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.389518    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"]
Oct 30 22:58:57.394697 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.394679    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:57.448931 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.448888    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"]
Oct 30 22:58:57.465975 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.465933    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baa85368-2f74-4239-be04-b278ca9d06cc-sys\") pod \"perf-node-gather-daemonset-mnnmv\" (UID: \"baa85368-2f74-4239-be04-b278ca9d06cc\") " pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:57.466135 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.465986    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/baa85368-2f74-4239-be04-b278ca9d06cc-lib-modules\") pod \"perf-node-gather-daemonset-mnnmv\" (UID: \"baa85368-2f74-4239-be04-b278ca9d06cc\") " pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:57.466135 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.466108    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/baa85368-2f74-4239-be04-b278ca9d06cc-podres\") pod \"perf-node-gather-daemonset-mnnmv\" (UID: \"baa85368-2f74-4239-be04-b278ca9d06cc\") " pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:57.466212 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.466136    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/baa85368-2f74-4239-be04-b278ca9d06cc-proc\") pod \"perf-node-gather-daemonset-mnnmv\" (UID: \"baa85368-2f74-4239-be04-b278ca9d06cc\") " pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:57.466212 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.466200    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4469q\" (UniqueName: \"kubernetes.io/projected/baa85368-2f74-4239-be04-b278ca9d06cc-kube-api-access-4469q\") pod \"perf-node-gather-daemonset-mnnmv\" (UID: \"baa85368-2f74-4239-be04-b278ca9d06cc\") " pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:57.567470 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.567434    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baa85368-2f74-4239-be04-b278ca9d06cc-sys\") pod \"perf-node-gather-daemonset-mnnmv\" (UID: \"baa85368-2f74-4239-be04-b278ca9d06cc\") " pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:57.567470 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.567471    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/baa85368-2f74-4239-be04-b278ca9d06cc-lib-modules\") pod \"perf-node-gather-daemonset-mnnmv\" (UID: \"baa85368-2f74-4239-be04-b278ca9d06cc\") " pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:57.567716 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.567493    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/baa85368-2f74-4239-be04-b278ca9d06cc-podres\") pod \"perf-node-gather-daemonset-mnnmv\" (UID: \"baa85368-2f74-4239-be04-b278ca9d06cc\") " pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:57.567716 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.567507    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/baa85368-2f74-4239-be04-b278ca9d06cc-proc\") pod \"perf-node-gather-daemonset-mnnmv\" (UID: \"baa85368-2f74-4239-be04-b278ca9d06cc\") " pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:57.567716 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.567569    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/baa85368-2f74-4239-be04-b278ca9d06cc-proc\") pod \"perf-node-gather-daemonset-mnnmv\" (UID: \"baa85368-2f74-4239-be04-b278ca9d06cc\") " pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:57.567716 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.567578    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baa85368-2f74-4239-be04-b278ca9d06cc-sys\") pod \"perf-node-gather-daemonset-mnnmv\" (UID: \"baa85368-2f74-4239-be04-b278ca9d06cc\") " pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:57.567716 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.567604    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4469q\" (UniqueName: \"kubernetes.io/projected/baa85368-2f74-4239-be04-b278ca9d06cc-kube-api-access-4469q\") pod \"perf-node-gather-daemonset-mnnmv\" (UID: \"baa85368-2f74-4239-be04-b278ca9d06cc\") " pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:57.567716 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.567585    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/baa85368-2f74-4239-be04-b278ca9d06cc-lib-modules\") pod \"perf-node-gather-daemonset-mnnmv\" (UID: \"baa85368-2f74-4239-be04-b278ca9d06cc\") " pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:57.567716 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.567585    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/baa85368-2f74-4239-be04-b278ca9d06cc-podres\") pod \"perf-node-gather-daemonset-mnnmv\" (UID: \"baa85368-2f74-4239-be04-b278ca9d06cc\") " pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:57.625387 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.625358    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4469q\" (UniqueName: \"kubernetes.io/projected/baa85368-2f74-4239-be04-b278ca9d06cc-kube-api-access-4469q\") pod \"perf-node-gather-daemonset-mnnmv\" (UID: \"baa85368-2f74-4239-be04-b278ca9d06cc\") " pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:57.707040 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.707001    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:57.745654 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:57.745629    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rfl76_8ca11674-8465-4bc9-b1a9-e2b99135d080/kube-state-metrics/0.log"
Oct 30 22:58:58.084680 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:58.084608    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"]
Oct 30 22:58:58.087815 ip-10-0-130-162 kubenswrapper[4473]: W1030 22:58:58.087778    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbaa85368_2f74_4239_be04_b278ca9d06cc.slice/crio-e908d49816dd14b321c5ecef21a8b40ff8d669efe861ff5359350ccf7ef65d37 WatchSource:0}: Error finding container e908d49816dd14b321c5ecef21a8b40ff8d669efe861ff5359350ccf7ef65d37: Status 404 returned error can't find the container with id e908d49816dd14b321c5ecef21a8b40ff8d669efe861ff5359350ccf7ef65d37
Oct 30 22:58:58.166272 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:58.166237    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rfl76_8ca11674-8465-4bc9-b1a9-e2b99135d080/kube-rbac-proxy-main/0.log"
Oct 30 22:58:58.585103 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:58.585077    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rfl76_8ca11674-8465-4bc9-b1a9-e2b99135d080/kube-rbac-proxy-self/0.log"
Oct 30 22:58:58.777408 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:58.777366    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv" event={"ID":"baa85368-2f74-4239-be04-b278ca9d06cc","Type":"ContainerStarted","Data":"c5e39b2526938cbace72cf0adcb2b8bc4fe301bdc68bba2a3d13aa2657d13a00"}
Oct 30 22:58:58.777603 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:58.777433    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv" event={"ID":"baa85368-2f74-4239-be04-b278ca9d06cc","Type":"ContainerStarted","Data":"e908d49816dd14b321c5ecef21a8b40ff8d669efe861ff5359350ccf7ef65d37"}
Oct 30 22:58:58.777660 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:58.777641    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:58:58.894667 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:58.894605    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv" podStartSLOduration=1.8945848349999999 podStartE2EDuration="1.894584835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:58.893153989 +0000 UTC m=+3957.129363334" watchObservedRunningTime="2026-10-30 22:58:58.894584835 +0000 UTC m=+3957.130794181"
Oct 30 22:58:59.008150 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:59.008116    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-9b5c6b74c-5w64w_3530dd2d-997b-48ca-9ffa-bfd102260075/metrics-server/0.log"
Oct 30 22:58:59.428143 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:58:59.427681    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-7l7hw_d494e8a5-5577-4ce2-af1f-cec218f37b50/monitoring-plugin/0.log"
Oct 30 22:59:02.377926 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:59:02.377894    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wfk5m_290ee42e-6821-4813-bb88-557f4e99946e/node-exporter/0.log"
Oct 30 22:59:02.794760 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:59:02.794715    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wfk5m_290ee42e-6821-4813-bb88-557f4e99946e/kube-rbac-proxy/0.log"
Oct 30 22:59:03.214136 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:59:03.214105    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wfk5m_290ee42e-6821-4813-bb88-557f4e99946e/init-textfile/0.log"
Oct 30 22:59:04.792583 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:59:04.792548    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k2tk6/perf-node-gather-daemonset-mnnmv"
Oct 30 22:59:08.692644 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:59:08.692610    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-xmc8p_5762586d-f5b1-4339-9c80-3e703eb00133/prometheus-operator-admission-webhook/0.log"
Oct 30 22:59:10.388896 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:59:10.388870    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c57bb4c94-x2drs_b599ef14-1c28-4c77-adbc-cb007513d00d/thanos-query/0.log"
Oct 30 22:59:10.810652 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:59:10.810624    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c57bb4c94-x2drs_b599ef14-1c28-4c77-adbc-cb007513d00d/kube-rbac-proxy-web/0.log"
Oct 30 22:59:11.233166 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:59:11.233126    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c57bb4c94-x2drs_b599ef14-1c28-4c77-adbc-cb007513d00d/kube-rbac-proxy/0.log"
Oct 30 22:59:11.304655 ip-10-0-130-162 kubenswrapper[4473]: E1030 22:59:11.304613    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nf7t4" containerName="extractor"
Oct 30 22:59:11.652471 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:59:11.652447    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c57bb4c94-x2drs_b599ef14-1c28-4c77-adbc-cb007513d00d/prom-label-proxy/0.log"
Oct 30 22:59:12.126236 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:59:12.126164    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c57bb4c94-x2drs_b599ef14-1c28-4c77-adbc-cb007513d00d/kube-rbac-proxy-rules/0.log"
Oct 30 22:59:12.546574 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:59:12.546552    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c57bb4c94-x2drs_b599ef14-1c28-4c77-adbc-cb007513d00d/kube-rbac-proxy-metrics/0.log"
Oct 30 22:59:13.601507 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:59:13.601469    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2mskp_9cc773b9-a6f5-48bd-9beb-11efeaae63d9/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:20.544871 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:59:20.544845    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6fc844489c-7k9tt_e2e69d3e-5768-4c46-8d25-d934422c8d4d/console/0.log"
Oct 30 22:59:20.986416 ip-10-0-130-162 kubenswrapper[4473]: I1030 22:59:20.986381    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-bj7lq_99a25eed-4eb9-4f87-8520-139ea86cf7ef/download-server/0.log"